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INTRODUCTIOKT 

: 

The purpose of this document Is to share various aptyfoaches used by Federal 
agencies to assess needs and measure training effectiveness. In all cases^ 
the evaluation extends to looking at the Impact (^f training on job behavior, 
in a number of the situations presented j the post --"cciurse evaluat^^oi^f lowed 
directly from the information oh employees' present 'kills and job ^ 
requirements developed before the training program Was put in place. 

This document is not meant solely to be an insplrvatton^l piece, as, in Robert 
Mager*s terms, "you really dughta wanna" do asses^rn^nt and evaluation. Nor i 
it designed to be an all-ihclusive how-to manual 'With e full set of 
instructions, forms, and guidance. We see it insl;e^d es serving chese 
purposes: 

- To provide ideas to readers on how different types of progrsTis 
can he assessed using various evaluation techniques. 

- To offer enough information to readers so ^b^t they can make 
decisions about the appropriateness of an «app>roach for their own 



setting, and can adapt examples of Instrmi^nt: s and design features 

k^* 

for their own needs. 

- To provide agency contacts who are willing td share their experi- 
ences with others, and thus to expand the PDr^ctlce of resource 
sharing and information exchange. 

Why were the particular agency efforts chosen? Tbief all have certain 
characteristics in common: Looking past the traihlag setting to measuring 
participant change; taking a systematic approach Cto gathering information; 
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enabling decisions to be made aboat the traininipr effort based on the 
evaluation results^ The approaches "worked" In the sense that the a^^enciiss 
were able to use the information iEtenerated for the intended purposes of the 
evaluations 

\. 

It should be noted that the emphasis in the descriptions is on the evaluation 
prorP5»R , not the results. In some cases, the results indicated that the 
traintnjsc was on target, while in others the evaluation indicated that changes 
were needed in the program, in the intended audience, or In the use of 
training as the means, or the sole means, for changie. The reader should also 
note that even for evaluation efforts which werie elaborately designed and were 
resource-intensive, certain aspects of the process fe.g., the questions asked; 
the Individuals contacted) may be applicable in other contexts and used in 
simpler ways. The matrix. at the end of this section highlights the features 
of the agency efforts. 

_ [ , ' f , 

Concerning the evaluation descriptions themselves, each begins with a summary, ^ 

followed by a general dascriptlon of the process^ After that is a brief 

analysis of the effort,; including a discussion of the resources needed and the 

advantages ^nd liraltationsi Finally, attachments are provided where 

appropriate as samples of data collection procedures and instruments. ' : 

In addition to the evaluation descriptions, several other types of information 
are provided in this package as references for the readers. Appendix A_ 
contains overviews of basic concepts in training evaluation, from the point * of 
view of two agencies whose internal documents have general application: ^ 
Department of Labor^s Empioyment Standards Administration* and the National 



Aeronautics and Space AdmlniRtratlon. 

Appendix B is a selected hibllbRraphy, While by no^ means a thorough iistinR 
of training evaluation books areJ articles, it can serve as a supplement to a 
training office's references. - • . 

Appendix C_ is a list of U.S. Office of Personnel Management contacts^ 
•including regional offices, that can be contacted for additional information 
on needs assessment and evaluation. 

Readers are encouraged to share their experiences on the subjects of needs 
assessment arid evaluation with 0PM, w^lch will in turn share useful 
information with others. 

•5 
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Agency 



Pnature 


Educ. 

(p. 6) 
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I; 
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performed 
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In 
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In 


In 


In (or 
other) 



Godes Follow-up method: Q = questionnaire; 1 = Interview; Ob = observations 

Follow-ap contacts: P = participant; S = supervisor 

Evaluators: In totally in-house training staff ; inPius = in-house 

training staff plus others in agency; InGon = agency assisted 
by contractor o 
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^ EVALUATION OF THE BASIC SUPERVISORY TR/iINING PROGRAM 

SUMmRY " 

The Department of Education evaluated Its basic supervisory tralnlnje course 
usln^ action plans and obtaining participants' perceptions of the use of 
skills from the course. Where possible, data was developed on return-OTi-7 
InvestT^anc (ROI)^ that is^ how much performance improved as a result of the 
training compared to the resources consumed^ In terms of dollars saved. The 
evaluation process also involved the training and use of volunteer line 
personnel I who had conducted the course, to collect Inforrhatibn after the 
course on participants* application of the skills they had learned. A 
consultative relationship was set up between evaluatdr and participant, such 
that the evaluator not only collected ROT data, but also helped the 
participants diagnose problems in applying skills. and assisted participants 1 
taking action to attain better results in using what they had learned. 



G^ERAL DESCRIPTION OF THE EVALUATIQi^ METHOD 



New supervisors throughout the Department of Education are required to take a 
40-hour in-house supervisory course, "Vorking with Employeeis . " The course 
particularly emphasizes comraunicatiohs skills. It also coveris isolving 
problems; improvinjf productivity; dealing with EEO issues; planning, 
delegating^ directing, and monitoring work; conducting meetings; and 
motivating employees. On-line since 1980, it is skills oriented and makes 
frequent use of case studies and role plays. Classes, which range in size 
from eight to fifteen people, are tayght by a carefully selected and trained 
group of volunteer supervisors from the agency. 

In order to determine if participants implemented skills lisarned in the 
course, if the course was worth the cost, and if the course itself needed 
improving, the training office (known as the Horace Mann Learning Center) 
developed an evkiaation methodology. The consultant services of the Carkhuff 
Institute of Human Technology were used. The evaluation process included 
these features: ^ 

l; Using the volunteer trainers to gather data. o 

2. Taking a diagnostic approach as part of the evaluation, to determine 
the reasons for the training being less effective than intended, and 
to assist participants with making more effective use of skills 
learned in the course. 

3. Developing return-on-investment (ROI) information, to show how much 
money was saved as a result of changed supervisory behavior. 



lU 



The first section b&low descrlhes the Ib^tlc behind the ROT approach. 
Following that are the steps of the' evaluation process Itself. 



Return-qn-Investment Concept 

ROI was defined as the ratio of performance achievement (output) to resources 
consumed (Input). The. ratio looks like this: 

( ' , ' ' ^' ' ■ 

ROI = ^ut fiut- bht a In ed « ^perfottfiance 

Input expended resources consumed 

The performance ach4evement part of the model was operatiotiallzed In th^ 

following way: Supervisors part Iclpatlnjs; In the training program caVi use the 

skills theylearn to attain a vast array of results (e.gi, fewer errors. 

Increased morale, reduC'ed bias, more efficient procedures) i The one common 

measure* that can be. used to reflect all types of results, and can' also be 

compared with resources consumed, is dollars sa^ted^ The use of dollars as a 

i " 

\ ._ j _.. ; ^ _ _ _ i _ _ : 

standard measure can be expressed as follows: : 

■ ' ■ ' , ~:y : 

Performance Achievements ~ Qollars** Saved + iii + Dollars Saved 
of Supervisor ^1 From Result "^1 From Result #n 

Performance Achievements « Dollars Saved + i.i + Dollars Saved ; 
of Training Group Ry Supervisor //L By Sup^rvi'sor ^^n^ 

■ _ _ > ._V . 

boiiar^^^^vln^s can be derived from costs associated with equipment, 
facilities, materials, and .personnel i Significant teductions in equipment, 

inateriais, and/or facilities costs resulting from training outcomes are rare, 

^ , •» _ . 

at least in the context of the Department of Education; computation of 

savings from these sources Is straightforward. 
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Personnel cos-ts havis proven to he the most common potential source of cost 
savings within the Department of Education. Reductions in personnel co$t8 are 
best reflected in the amount of time saved. For example, fewer errors as a 
result of employes coaching means that less time is spent correcting work. 
The time -saved from the result, of fewer errors can be computed, then 

translated to dollars using the hourly pay rates ; of all • the staff whose time 

^ - " ■ _ •. ' 

is saved . „ ' -a 

Increases in supervisors' skill UBe\can result, too, in substantial savings 

through the identification of tnisuse of funds allocated in the "f orbs of loans, 

•grants, ^rid contract^; e.g., problemrsolving^ and communicatio.n skills\dould be 
' :, ' " '_ _ . . _ ^ _ _ 

useful in identifying and helping coirrect reasons for not spotting misdirected 

., . ' „ _ * ' 

funds. Information on this type of savings is more difficult to obtain and 

veri'fy but when uncovered is usually quite significant. . 



The -resources ronsnmed part of the model was developed using this, approach: 
To be used in the ROI formula*, resources consumed must be expressed in ^ 
dollars. To achieve individual and organizational results, supervisors 
consume, both training resources^ pius whatever additional resources they 



require to make changes on the ioh. 



Training 

Resources = Resources + Implementat i-on 
r onstrmed C onsumed Res ources 



Costs for training and costs for implement at io'^^'are derived from the same 
categories used to compute dollar sayings (i.e., facilities, ^^tfipment,. 
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raateriai:r. , ^.tid persohnisl). The major cost for both training and 
implementation is personnel. In computing personnel costs, the time involved 
for all personnel is accounted for; For example, in computing training 
personnel costs, both the cost of instructor time and the cost of student time 
are included. 

An example of the computation of return-on-investment for one supervisor and 

: _ \ 
for an entire class is given in Attachment 1. 



^Evaluation Process 

The evaluation of Education's supervisory course, 'Vorkihg with Eraployiees," 
involved these steps: developing the evaluation process and instruments; 
training volunteer evaluators; setting the stage in the class; gathering data 
and following up; and analyzing findings and developing recommendations. Each 
is described below. * 



Development of process and instruments - The evaluation process described here 
was developed after an earlier approach to follow-n.tp with participants proved 
i^workable. The first effort involved forms which the participants 
(supervisors) were to use to tally the application of skills learned in 
training and to define the results obtained; these were to be filled in daily 
or weekly and to be collected after three months. However, supervisors tended 
not to recognize when they were using a new skill and not to use the forms to 
track changes they did recognize. 

The revised process addressed these difficulties by relying on the evaluators 
.to work with the participants after the course to help them d(gtermine when 



they were using course skills and the effects of their use; to diaj^nose areas 
for potential behavioral change; and to determine how the supervisors could 
best use the skills to achieve change, A structured interview format guided 
the evaluator in obtaining the information from the participants about sskill 
application, in doing problem diagnosis, in helping plan for successful skill 
application in one area of emphasis, and .in developing benefits information 
leading to ROT analysis, if appropriate. ^(See below for more detailed 
explanation.) 

Training of volunteer evaluators - In order to carry out the labor-intensive 
evaluation process, the course coordinator (who was also the evaluation 
coordinator) and one of the consultants trainied four of the volunteer trainers 
as evaluators. However, this approach could be used for staff evaluators who 
did not actually teach the course but were willing to leairn the course 
curriculum thoroughly. 

the multi-phase training included attending the supervisory course in full and 
receiving instruction in training iskills, in evaluation, and in consulting. 
The volunteers observed an interview of a participant conducted by the 
evaluation coordinator, then were observed themselves when they conducted 
their first interview. After their first interview^, meetings and debrief ings 
were held, individually and as a group, to discuss the interview process. 
(Note: Each volunteer evaluator is now expected to follow up with a total of 
three course graduates afte each course session.) 

Setting the stage in the class - Participants were told at the end of the 
course that they should identify one course skill in particular that they 
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wantedv to develop and apply on the jobi They also were informed that the 
Instructors wbiilil follow up with them to determine the value of the course in 
chanjE^ing job behaviors and to assist them with any difficulties encountered In 
putting the course skills to work. (Details of the ROI were not explained to 
them until the actual interviews.) ' 

Data gathering and follow-up activities - Approximately two weeks after the 
end of the course^ each participant met with an evaluatbr who sought to find 
out if the supervisor was using course skills on the iob and, if so, which . 
ones and how frequently. The evaluator also obtained the reasons for using or , 
not using the skills learnied (see Attachment 2). In all cases supervisors had 
used some of the skills wlth^ In most Instances, positive results. Where 
supervisors had nb^ little ^ or poor results^ the evaluator worked with them to 
address the reasons. 



The evaluator then assisted supervisors to sielect one skill in which they 
particularly wished to improve and to develop a plan for behavorial change 
using that skill. Finally, the evaluator, also in concert with the 
supervisor, developed ways to measure the results of the new skill 
application. For example, supervisors wishing to improve their skills in 
communication might be encouraged to try using reflecting skills three times 
in each meeting with an employee and to note if the narer understanding that 
resulted led to time savings by virtue of eliminating the need to redo work 
and to meet again. 

The evaiuators returned to the participants about two months after the Initial 
post-course discussion, to obtain specific results of attempted change for 
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translation Into cost savings. At that tltiie the evaluator provided additional 
advice to the participants if the situation warranted i (See Attachment 1 for 
examples of participant changes tnade and resaiting savings.) 

Analysis of findings and deArelopment -o£JE±nBt_recomme^ - the Training 

Director and Department management, rapidly becoming too sophisticated to 
accept "smile indexes" of success, are now demanding evidence of increased 
productivity from training. The ROIs developed, even when calculated 
conservatively, provide firm evidence of such results. 

The data and recommendations generated have bieen and will be given in the 
future both in oral and written formats to the Training Director and the 
Deputy Under Secretary for Management. The first ROI computed justified the 
decisions to continue using a tailored in-house course versus an outside 
course, in-house volunteer trainers rather than consultantis , and a course six 
days in length rather than a shorter one. 

Analysis of the findings also clearly indicated improvements to be made within 
the course (e.g., creating a more problem-solving context for communications 
skills and using more case studies), which were then incorporated in the 
course in hopes of increasing the ROI in the. future. 

the ROI evaluation model will be Implemented on a yearly or semi -yearly 
schedule. Should the ROI drop. It could be Increased by decreasing the 
implerientatlon costs (e.E;., shorten the course) or increasing the savings 
(e.g., focus only on skills that yield clear savings). 
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ANALYSIS OF EVALUATION METHOD L 

Conditions for usea , 

- Management support to allow for the evaluators' and the course graduates' 
time. 

- A training office concerned about obtaining specific evaluation data 
and willing to support the evaluation coordinator's efforts. 

~ Evaluators who are knowledgeable in the process and content of the course. 

- For volunteer trainers, training as neceissary in interviewing, problem 
solving, and consulting techniques. 

- Also for volunteer trainers, the motivation to continue working with coarse 
participants and to carefylly gather information \^ich will be useful for 
evaluative purposes. 

- An evaluation coordinator able to oversee the efforts of the volunteer 
evaluatorsi 

-A system for rewarding volunteer tralner/evaiuators (examples are -cash 
awards; letters of commendation) and their supervisors. 

- Interview guides for the evaluators. 

Resources required 

The Individuals involved in the evaluation are a coordinator (who is a full- 
time training staff member and who may also carry out fbllow-nip activities as 
we] 1 as oversee the efforts of the volunteers) and volunteer evaluators 
(volunteers in the sense" that evaluation is not pfirt of their usual job). 
Time is the main resource needed. The coordinator may spend up to three staff 
days designing and conducting training for volunteers who need interviewing 
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and other skiils, then conducting interviews (olfterSred by the voiunteeris), 
observing their interviiws, and debriefing the -viSHTiteers. The evaluators in 
turn may need about a day's worth of training^ imclXJdlng formal sessions, 
observations of interviews, debriefings, and conflactlng an Intervrew 
Independently while being observed. 

For the time required to evaluate one course** tlies coordinator would need about 
two staff days In the administration' of the precise and preparing a final 
report, while the evaluators would need about a ^taff da^r to conduct three 
interviews each and brief the coordinator. The CiOOfdinator tna^ also find it 

necessary to put more time into training and de"br^tftfing the evaluators, 

> _ 

depending on their skill level.' . 

p 

1 _ - - - J - 

No statistical knowledge is -required on the part of any of the individuals. 
Nature of inforination pt^Qd4jieed 

As originally conceived, this evaluation process wats designed to collect 
return-on-lnvestment information - the costs vs. tlxe benefits of the training 
course. The interview process also provides InftiinTiation on: a) problems that 
course participants! routinely face on the job, actii the general context in 
which they have to perform their work; b) problenis course graduates experience 
specifically in practicing course skills on the ::io1i and the causes for such 
problems (e.g., lack of supportive managerial cWra^te; lack of understanding 
of how to use the skills; lack of comfort or exp^rtence in using the skills); 
c) reactions to the course and the trainers, fro*i the vantage point of being 
back on the job for a period of time. 
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Advantages 

- Provides suppdrtive data for the course,^ for the training function toigtve 
to top management,^ 

- Motivates participants to continue using skills they learneiV, as they are 
helped to see clear evidence of results^ 

- Provides help to participants experiencing difficulty using course skills 
back on the job, thus increasing transfer of learnings to the job and 
thereby the ef feet ivenes s of the course • 

- Provides additional information to the trainers on the relevance of course 
skills taught and the context in which they are used, 

- Develops an ongoing consultative relationship between course graduates ..and 

the evaluators, thus giving the participants a source to turn to fc^ 

. ' 

assistance if problerns or questions arise, 

- Motivates course instructors as the information on positive results from 
the training lets them see firsthand the impact of their efforts, 

Lifflt tat ions ' . ' ' 

-Participants' own abilities to recognize when they have been applying skills 
learned in the course, along with the training office's having to rely on 

participants' judgments about the results from using new skills. 

- - - - ^ ...... . 

- Heavy reliance on interviewers to draw accurate information from the 

participants and assist the participants in developing the required data, 

- Absence of v7ork tracking arkd measurement systems as isources of datai 

^ Limited time that can be asked of volunteers to gather data, and that can be 
asked of participants to provide information, (Experience has shown^ 
however^ that participants have been cooperative in the evaluation effort^i) 
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- Potential bias of instruct ors* interviewing participants and seekinrc 
examples of changes attributable to training, in order to prove course 
worth* 

It Should be noted that, despite these limitations^ even when conservative 
estimates are used to derive ROI figures ^ impressive results have been shown 
(see example in Attachment !)• Indeed it is wise to make conservative 
estimates to offset criticism of the data's siibjeqtivity, 

^Ref ere ncft 

Lticy McElij^ot . 
Horace Mann Learning Center 
UiS. DepartTjent of Education 
3700 Donohoe 
400 6th Street, SW 
Washington, DC 20202 
(Phone: (202) 245-2481) 



2' ' 
^ 1/ 
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RETURN -ON -IN VE STM©I T 
CALCULATIONS 
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Keturn~on-Investment^alcula^OftS^ 



An example of the computation of return-on-inves tment for one- supervisor Is 
the following: 

Result #1: o Skills used - Interpiersbnai and time managetnent 
b How used - To develop procedural changes 

b Result - Time reduced one-third for each technician 

9 _ _ ^ 

o Total ^ avln g^g- - 3,000 technician hoars per year 

$5;50 per hour 

Result 1^2: o -Skill^usgd - Controlling 

o How used - to establish phone moiiitoring system 

o Result^ - 10 logged phone calls per day eliminated 

at 3 minates each; 5 non-logged phone 

calls per day eliminated at 10 minutes each 

o Total savings - 333 hours of clerical time per year 

P $5.50 per hour 



Performance -Dollars Saved + Dollars Saved = $18,331 

Achievement: From Result #1 From Result 

($16,000) ($1,831) 
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training 
Resources: 



o 48^ hours of supervisor training 
time 3 $15*00 per hour 

o 18 hours of instruction time 
$1^.50 pier hour (a total of 
144 instructor hours were" 
invested in 8 participants^ 



Implementation o 16 hours of supervisor time per 
Res-ourceSv; . 

year 3 S15.00 per hour 



o 80 hours of technician time per 

year ^ $7.50 per hour 
o 32 hours of clerical time per 
year ^ $5«50 per hour 
+ 



Resources 
Consumed : 



Training 
Resources 
Consumed 
($1035) 



Traplieme ntat ion 
Resburcies 
Consumed - 
($1016) 



$2,051 



ROI: 



$18,331 
$ 2,051 



« ROI of 8.9 to 1 



In computing the ROI, it should be noted that a standard period of time must 
be selected. In the example, the time period selected was one year. 
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The computation for a j^roup of el^^ht course participants can be done as 
follows: 



Results from 
Each Supervisor: 



Performance 
Achievement: 



Supervisoxr #1: $37^125 
Supervisor #2: $18^331 

Supervisor #3: $ 3,125 

•* 

Supervisor ^A: $10^000 

Supervisor ^5: None 

Supervisor #6: None 

Supervisor #7: Not available 

Supervisor #8: Not available 

Hollars Saved by = $68,581 

Supervisors 1 to 8 



Training 
Resources: 



o 384 hours of supervisor time 9 
$15.00 per hour 

o 144 hours of instructor tttne P 
$17.50 per hour 



Imp 1 erne nt at ion 
Resources : 



Resources 
Consumed : 



Supervisor 


#1: 


$5,000 


Supervisor 


#2: 


$1,016 


Supervisor 


#3: 


$ 400 


Supervisor 


H: 


$1,500 


Ss^ervisor 


#5: 


$ 175 


Supervisor 


#6: 


None 


Supervisor 


#7: 


Not available 


Supervisor 


#8: 


Not available 


training 




Implementation 


Resources 




Resources 


Consumed 




Consumed 


($8,280) 




($8,091) 



= $16,371 



ERIC 
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Roi: $fiR.581 i ROI of 4.2 to 1 

$16,371 , 

Two points heed to be made in rep^ard to the above example. First ^ 

trainln^^ participants were used for the cbmjjUtat ion of the ROI ~ even the two 

participants for whom results were^ not available. If these .two .supervisors 

are eliminated from the training resources equation, then the ROI improves to 

4.6 to 1., Second, the life cyclie model of computing training costs indicates 

that there are three distinct stages: 

• ^ * S - 

Start-up stage - When the program is being developed. 

Transition stage - When both new programs and old programs 

are being run. 

Steady-state stage - When the program is fully operational 

and changes are miniraai.^. 



The above examples have been based on a "steady-state" stage; If it were 

desirable to do so^ start-up costs for design and deyeioiiment would be 

_ ■_ _ y 

determined and a share allocated to an individual trainee and/or a training 

group; this would have significantly reduced the ROI for the initial group 

cited but not for later groups where Start-up costs no longer apply, 

_ y . -nr - ■ . 

(Alternatively, start-up costs could be amortjtzed over the length of the *^^use 
of the course, thus minimizing their impact on any one course offering. This 
is prbbably a more realistic approach and more closely parallels private 
sector business practices.) ^ 



ATTACHMENT 2 



FOLLOW-UP INTERVIEW FORMS 
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Providing Assistance to Supervisors Who Took the Course: 
"Working With Eiployees" 

(For trainers interviewing supervisor trainees) 



Is the sopervisor using the coarse skills ? (Page 2) 



Supervisor is not using 
Coijirse skills 



Supervisor is using 
Course skills 



, Identify the reasons. 

(Page 2) 



. Address the reasons, 



■ Develop a plan, 



(Page 3) 



(Page 4) 



1. Indicate frequency of i^e. 

(Page 2) 



2. ' Identify reasons for use. 



Supervisor his results 



1. Identify the effects. 

(Page 5) 

2. eoiupate the benefits. 

(Page 5) 

3. Consider an additional plan, 

. (Page 4) 



o 
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(Page 2) 



Supervisor his hb results 



1. Diagnose the situation. (Page 6) 
Identify problem skill 
Identify reason for use: 
- Problenjneel 



Pinpoint problem: 

- What's happening 

- What" want to happen . 
identify reasora for problem 

- Reasons 

- Suggestions 

2. Refine the plan. (Page 4) 



2: 



Providing Assistince To Siipervisoi^ 
The Course: "Horklng. with Employees" 

Supervisor Position -- ' Date 

Interviewer Unit . '^"^^'^^ 

Level 

Tdtir iise of -course skills : 



indicate how often the supervisor has used or is using 
the course skills on the job listed below: ' 


For each skill write below a 
reason for the supervisor 
using (e.g., action plan 
developed, iomiediate need, 
special interests) or not 
using (e.g., not enough 
time; not motivated, don't 
understand the skill, not 
relevant; no opportunity) 
the skill. 

Reasons: 


Course Skill 


More than Orice a Less than Once a Less than Not at 
once a day once a day week once a all 
day but tnore^ week 
than once 

a week ; 


1, Communication 

2. PRD 




1. 




2. - 


■Solving perform- 
ance problems 
.Setting work 

expectations 

3. EEO 

i... Planning 

.Short-term 
.Long-term 

5. Directing 
Jaklng assign- 
ments 

.Conducting 
•" lee tings 

6, ControUiiig 


• 






h. 




5. , 




6. 


.Monitoring work 
.Motivating 


it 

-2- 
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Developing a Plan For The Supervisor Who^S not Using 
The Course Skills 



The supervisor is not using any ol- the course skills. Below are suggestions for addressing sout of 
the reasons for the supervisor not using the skills. Review these reasons with the supervisor aid 
develop a plan by filling in the rest of this page and the following page- 



Reason 



1. No time available, overloaded with work. 

2. No action plan from the course. 



3, Do not understand the area. 



4. No need at this tine, 



5. Not motivated, 



6. No support from superior. 



I. 



8. 



Suggestion 



1. Try a very small part. 

2. Develop an action plan. 

3. 'Review the material. 

4. Plan improvements, 

5. Try a very snail part that Interests you, 

6. Meet with superior and try a snail part, 



7. 



B. 



Skill 



-Reason • 



-Sa■g■gest4on- 



3C 
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Developing a Plan to Use Course 
Skills on the Job 

in the spaces below, develop a plan for the supervisor to use 
one of the course skills on the Job. 



Indicate the skill to be used: 



Indicate the reason the skill is to be used: 



If possible, tie the reason to a perf ormance problem 
(individual or unit). 



Develop the plan. 

Jfhat..: 



With Whom: [ Self /employees (no.)] 



Steps 



EKLC 



Time Frames 



-4- 



Recording the Resaits of IlBing the Course 
Siciiis on the Job 



indicate _below the results the supervisor has had in using the course skills on the job. Also deterttine 
the benefits based on the results. 

Effects: 



What Happened? To Bhbni? How Often? 


Average Time X Niiiiiber of - Time To ■ 
Savings Instances Plan Benefits 


■ .)»,>v n-i». ».«,"<. nr., 1 . 1... '1- ihii n, ,.«„*••'■> ,,......._„., 

i 


1 

} 

1 . ' 



1, Refer to page 4 for developing plans for other course areas, 

■05 

2. Re'er to page 6 for problens that you may have. • . 

-5- 
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Diagnosing, the Problem for the Supervisor Who 
Has No Results In Using Course Skills. 

Ask the supervisor the questions below in order to address 
the skills for which the supervisor does not have results. 



1. with whait skills. are you having a problem? 


2. What Is your reason for using this skill? 


A specific need or problem: 




Other 




3. What are the effects that you are not achieving? 


i What is no t happening? 




What do you want to hapjpen? 




4. What are the reasons for the problem? 


Here are some examples. 








Re a sons : 


Suggestions : 


• Your understanding of the 
area ^ 


1. Review the material. 


2. You are looking for inappro- 
priate effects . 


2 . Revi se purpose . 

'\ 


3. Your superior is not supportive. 

4. Employees are not motivated. 


3. Meet with your superior 
and/or simplify plan; - 

4. Meet with employees 
and/or simplify plan. 



3e 



Diagnosis Cont ' d • 



What are the reasons ? 



Your reasons" 



Sagges tions 



5, If revisions in your plan are required, use page 4 



37 
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TKS. GENERAL ACCOUNTlSf; OFFICE 



EVALUATICW OF TtTE SKILLS FOR PERFORM/VNCE 
AND CAREER DEVELOPMENT COURSE 

SUMM\RY 



GAG developed a course to train their: auditing staff in interpersonal problem- 
solving processes and to enable ind^ividuals to use the skills learned in 
various agency--6pecif ic settings. An extensive Evaluation was carried out to 
assess participant reactions to the course and knowledge gained (course 
process ipeasures) and participant application on the ;job of skills learned 
(course product^^easures)^ Various control groups were used to determine, fop 
instance; the influence of the course on behavior ; relative to other 

__2 __ . ^ ' 

interventions such as "sensitizing" individuals to the sub^ept "raattari 



Results of the evaluation were used for a variety of purposes, including 
modifying offerings of the coarse for other-GAO staff , deterraininf^ use of- 
skills in the ^:gency, and showing the value of internal evaluation. 
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GHNERAL DE9CRX^TI(»^ -OF- THE^S^UMHim METHOD- 



buring the late 1*^70*8, the General Accounting Office (SAO), by direction of 
the Comptroller General, developed a training course called Skills for 
Perforraance and Career Development (SPCD). Its purpose was to impart generic 
interpersonal cominunicat ion skills which could be used in a variety of 
I counseling situations* The audience .attending the course included, almost all^ 
. GAO auditors/evaluators and many support staff, numbering over 3000 
individuals. 

a 

The objectives of the four-day workshop (which was offered over 200 times) 
were to enable participants to: 

1; Learn about effective and ineffective communications skills. 
/ 2i Practice the effective skills. 

3. Apply the- skills to performance coaching and counseling 

(including personal problem identification)^ career counseling, 
performance appraisal, and small group problem— solving situations. 

Using written materials, lectures^ large and small group discussions, role- 
play In js;, and video-taped illustratidns ^ the workshop was designed to use 
participants'" own expe|:iences and to ensure direct applicability of learning 
to the GAO organization. The participants continually received feedback on 
their competencies through audio and video replays;. Part of^the learning 
process was participants' keeping dally Logs on what they were learning- and 
how they would apply that to their own jobs. ' 
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in order to linplemisnt the SPCD course on the massive scale desired, external 
consultants and internal personnei were trained by j^olnja; throu^^h the course, 
then by receiving instruction on how to conduct the course themselves and how 
to administer the evaluation instruments. Classes were given In Washington, 
DC, and in various regional sites. While an observer from the central 
training staff watched many of the sessions ^ ho attempt was made to 
standardize the specific course content or presentation style. Thus while the 
basic SPCD skills were to be covered each time^ the implementation of the 
course varied from session to session* 

f 

The evaluation design for SPCD was starts after the course was being 
delivered. Both the course "process" (in-course training) and "product** (use 
of skills in the GAO setting) were assessed^ the process was assessed at two 
levels ~ participants' reactions to the course and the amount of irifbrraatioh 
gained during the course. The product or outcome was measured by looking at 
behavior change after the course. Because of the timing of the evaluation 
start^Pi the results were not intended to change the original SPCD course as 
it was being offered. Rather the purposes of the evaluation were to: 

1. Indicate changes which should be made in* other SPCD programs, 

( _ ' _ 

e.g., one for non-auditors and one for new auditor staff. 

2. Assess the .impact of the program on the awareness and use of 
interpersonal skills In the agency (as a prelude to intro- ' 
ducing performance appraisal and coaching systems into the 
agency) • - ' . - ' 

3. Reinforce the agency's perception of the importance of the 
skills taught. 
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A. Demonstrate the feasibility and value of internal evaluation - 
projsjrams. 

The following describes the evaluation effSfts for both the process and 
product of the SPCD project, including the means for collecting and analyzing 
the datai 

Evaluation of the SPCD "Proeees" 



Design - In order to understand the effectiveness of the training process, 
fcmr groups were used - an SPCD group, a Sensitized Awareness Group (SAG), a 
cohtrol group, and a post-test only group. The characteristics of each were 
as follows. 

The SPCD group took the p^e-test at the start of the course, coinpleted the 
four days of training, and took the post^test at the end of the course. The ^ 
Sensitized Awareness Group (SAG) took the pre-test, received /sensitizing" 
instructions that made participants aware of interpersonal problem-solving and 
communication skills without providing training in the skills, and -toolc the 
posts-test four days later. The control group took the p^e-test and: four days 
later, the poet-test ; this group was used to assess the/ stability of the 

asuring instruments over time an-i repetition. The post-only group did not 
take the pre-tcst but did complete the full four-day' course and took the postr- 
test at the end; they serv^ as a comparison group /for the SPCD group to see 
the effects of taking the pre-test on the impact of the course itself. 

Data collection^strum^ts -"^The Instruments lised to evaluate the course 
process were developed based on the course objectives ,, the cmethods used to 
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accomplish those objectives, and the design of the course, the instruments 
(pre-test and post-test were identical) included a Belie? Questionnaire, an 
Information Assessment Quest lonnairie, and an Ohservation Exercise, as well as 
a Course Evaluation Form. Their characteristics are summarized in the 
following chart.* 

Data Collection Instruments (SPCD "Process") 



Instrument 



Information obtained 



When jgiven 



Belief 

Quest ionnaire* 



Inf drmat ion 
Assessment 
Quest ionnaire* 



Observation 
Exercises* 



Course 
Kvaluat ion 
Form 



- attitudes about course content 

- attitudes about, status of 
comiminication skills at GAO 

- attitudes about status of personal 
communication skills 

- measures of knowledge and 
comprehension of, SPCD skills 

r ™^M"r^s of ability to apply 
SPCD skills 

- measure of ability to recognize 
SPCD behavioris in sitmilated 
CAO setting 

- irnpressibns abcuv instructor 
competence 

• Irapressidris about instructor effort 

- impressions about value of specific 
aspects of the course 

- impressions about overall value of 
the course 



pre-test & 
post-test 



pre-test» & 
post-test 



pre— test & 
post-t est 



post-test 



*See Attachment 1 for examples of the first three instruments. 



'4i 
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Pata analyses - The course process evaluation used over 2000 sets of cbmpleted 
data"' For ms (frditj the four experimental groaps)i Answers were computer- 
tabulated ^ using a specially designed program. For each participant the mean 
of all responses related to a particular type of information, or subscale, was 
computed and these means were uSed for analysis purposes. For instance, the 
effect of giving a pre-test on post-test performance was examined, comparing ^ 
the post-test only group to the SPCD group. An important analysis was looking 
at the effects of the course itself on knowledge, attitudes, and skills. 

W^ile statistically significant differences were shown for those who took the 
course versus the comparison groups, the results could have been due to the 
large sample size; the differences in the numbers themselves were not very 
large and hence did not seem of practical importance. Therefore the 
evaluators did further analyses to assess- the influence of the sample size. 
Results pointed to one particular area of practical significant difference 
between those who took the course and those who did not (ability to apply 
communication skills in simulated interpersonal encounters^ from the \ 
Observation Exercises). Finally, the results of the course evaluation form 
were calculated. These results were also correlated with res|5onses on the 
other course process measures, to determine the relationship between how 
participants felt about the course and how much knowledge, skill, and attitude 
change occurred. 

Evaluation of^^^hT^lTCD "Pr.oduct" 

Design - Because the goal of the SPCD program was .to improve the interpersonal 
prohiem-solving'and communication skills of participants, efforts were made to 
assess the tmpact of the course on behavior in the GAO environment. To do so. 
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two types of data cbllectlbn processes were used - a Follow-up Questionnaire 
and an Observation of three audit teams. 

Data collection instruments and data analyses: Follow-mp Questiohhaire - A 
Follow-up Questionnaire was sent to GAO employees eligible for the SPCD 
program, whether or not they had attended the training (so that comparisons on 
kncfWledge and skill could be made betwieen course participants and those who 
had not attended). The questionnaire was sent out about nine raonthis after t^he 
beginning of the training implementation; In addition to demographic data, 
^the questionnaire consisted of three major sections; The first requested 
self-reports about the frequency of attending meetings, the use of 
interpersonal skills, and interpersonal' effectiveness , in five GAO 
organizational settingsi These settings, esgi , audit team meeting, agency 
meeting, personal problem-solving session, were selected because they occurred 
relatively often, were more or less mutually exclusive, and covered most of 
the interpersonal, job-related encounters a HAG employee was likely to 
experience. 

The second part of thefquestloiinaire, answered by SPCD participants only^ 
requested self-reported , awareness of comraunlcatibn behaviors . used by 
themselves and by others^ and perceived course benefit. 

i» 

The third section asked Tespondehts tb^ 

; ■ / 

•s - - -- - - - - _ _ _. _ _ _ . 

simulated Interpersonal problem-solvinjEj situations^ by having them check what 
they? felt would be ,an appropriate response to sample employee statements about 
problems. (See Attachment 2 for Fbllbw-up Questionnaire statetnents. ) 
Definitions were provided to, respondents on the various terras used in the 
questionnaire. ^1 ' 
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Data from the questionnaire were sutninarlzed by Ibokinj^ at all SPCD 
participants, at all ndn-SPCD respondents, and at SPCD participants who had 
taken the^ course six mdhths previously. Data from thia last grdup was directly 
^-iJinpared to xrespdnses of the non-SPCD group, as the non-part Iclpants were 
Instructed to-respond to the questionnaire based on their experiences for the 
last six months only. Analyses included frequencies with which the 
respondents reported being in the five organizational settings and, for each 
setting, siBiimaries of the kinds of coramanication behavior, the respondents 
reported usingi Statistical analyses were also done to compare responses 
among the three groups. 

Responses on the~"seif -asses sment—of- interpe^ effectiveness were 

summarized to show the extent to which course participants saw themselves as 
changing in effectiveness in different organizational settings. Perceptions 
of course benefits arhd awareness of communication behaviors were also 
tabulated in a straightforward way. 

The third section of the Follow--up Ouestionnaire lised the same five items as 
on the pre- and post-test to assess the ability to apply communication skills 
to -simulated interpersonal encounters. SPCD participant responses were 
compared to responses of non-participants to determine statistical 
significance. 

Data collection instruments and data analyses: Observation of audit teams - To 
ohtRln fflT^rrM>n -^t^ aKpnt ^the -use of communica tion_jkills in the GAO 
environment, three audit teams were selected for observation: Onie in which 
the team members had completed the course; one in which the team members were 
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aware that commanicatlori skills were being obsieryed but had not taken the 
course (SAG); And one in wtiich team tnethbers had not taken the course and did 
riot know why they were being observed (control). Observers were trained in 
observation techniques and srCD skills. Structured forms were used during the 
observations to capture S'ach information as type of meeting being observed and 
specific behaviors exhibited. Observers' were to look for five positive skills 
taught in the course and five negative behaviors which, if present, would 
impede the interpersonal problem-solving process from odcurring. Observers 
contacted the audit teams to find out about scheduled meetings^ with the audit 
group letting the observers know about any meet ings scheduled on short notice. 

Analyses were made of the frequency of meetings of different types among the 
three groups observed, as well as the rate of behavior occurrences per minute, 
for positive and negative behaviors^ The types of behaviors observed were 
compared with the results of the Follow-up Questlonnairer 

Pijnal Report 

A report covering all aspects of the evaluation - process^ findings, analysiis, 
recommendations, instruments - was prepared and preserited to the agency's top 
management i Abbreviated versions of the report were presented to other levels 
of management and the agency as a whole. The audit teams used in the follow- 
up observation, the course instructors i and the others involved in the project 
were debriefed and Inf orTnS"^rThenrei^^^ Result s "wlFe~Used t d^Mesl gn^'tlTe" 

SPCD course for later audiences and to reinforce SPCD skills in subsequent . 
managerial/supervisory/executive development programs and during the 
implementation of the agency's performance management system. 
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ANALYSIS OF EVALUATION METHOD 

Gonditlohs for u8e 

Support by management for the time to design and implement an evaluation 
of such extent. ' 

- Involvement of evaluation staff early in the process of desij^ning the 
course. 

- Resources which can be tapped f or assistance in , questionnaire deisign, 
sampling I and analysis techniques. 

- Cooperation among those consulting on the evaluation, those conducting 
the evaluation, arid the instructors, as well as the participants (and 
their supervisbrij) • 

- Instruments specifically designed for the agency's environment and 
based on course objectives. Lead time to pilot-test instruments^ 

- Staff which can be made available and trained specifically to help with 
a large-scale evaluation, or ability to hire outside consuitantSi 



Jies purees - required- « 

The time of the evaiuators to design and implement the evaluation process and 
to analyze the results, is the primary resource required. Also, if sampling 
of participants and statistical analyses of data are to be done, expertise in 
these areas needs to be available; if the training staff does not have this 
Will ,'"rrid f vi'di^^ 

assist the evaiuatorsi The statistical analyses are straightforward for the 
most part and standard statistical approaches can be modified for specific 
data analysis needsi A small amount of time is needed on the part of 
instructors and participants In fulfilling the requirements of the evaluation 
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process* If observations, are included in the desljscn^'^ihdividuals need to be 
trained to perform this task with consistency and reliability. If a large. 
satDple Is to be iised^ a computer program to process the data can save 
considerable time In the long run, 

Na^tiire of^ lnf<>CTiat4ori^ produced 

With the dGislgn used at GAO, evaluative data are produced at three levels - 
attitudes of participants about the usefulness of the course, knowledge 
changes, and use of skills on the jobi As a check on the effect of the course 
itself, comparisons can be made between groups of participants and certain 
comparison groups (e^gi, those without the training; those sensitized by 
receiving information on the subject witliout formal training )i The evaluation 
instruments also produce information on how respondents felt about using 
communication skills themselves and the Impact that that has had on their 
perf ormancei The information is derived in different ways - self perceptions, 
TM^rltten tests » and ohservaf ions by trained individuals who were not GAO 
employees* The information produced is quantitative and includes descriptive 
and inferential statistics- 
Advantages^ 

- Using comparison groups as a means to control for certain variables 
(such as the effect of taking a pre-course test on the course's 
effectiveness) allows stronger conclusions to be drawn. 
Obtaining evaluation information from different sources and by 
different data collection processes enables verifications to be made. 



i 
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Limitations 

- The ne ted for meiasurement expertise to design and ' Implement instruments 

■ / • - 

and tabulate results. 

- Extensive lead time and resources required to carry out a large- 
scale -evaluation (saraplinR would diminish this need to some extent). 

-With the evaluation process begun after-the-fact, limited use of the 
results for modifying the ongoing training (can be used in. future 
offerings and offerings for ^her audiences). 

Refere^nce 

Dr. Steven M. hfediin 
Room/ 7840 
GAO ;Buiiding 

441 ;6 Street, NW ~^ 
Washington, DG 2D548 * 
(Pfiiones (202) 272-3162) 



EXAMPLES OF Mtft eeLLECflGN 
INSTRUMENTS USED ^ 
(SPGD -PROCESS") 



/ 

1- - 

/ '- ' ■ 

^ I ■ ' 



IDENTIFXCATIOII CODE: (t€«t £wr digit* of yoar home 
telephone number. Pleaie ri^cord here and in your 
notebook for future reference • ) 

DWISION/OFFICE/REGION (Itxiert the neiae of your 
division^ office g or region in the ipace balow and 
insert. the code number in tht 2 boxes to the left ai 
given by the instructor.)^ 



eSSREKt GB GRADE LEVEL (Insert the number (1H5) of 
the category in the box.) 



1. 


Grade 


6 or below 


2. 


Gride 


7 or 9 


3, 


Grade 


11 


4. 


Grade 


12 


5. 


Grade 


13 or 14 


6. 


Grade 


15 or above 



Card 1 
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? leixors Qotsiiomkut 

"^Iftti i«' « quctciottiUiri _«^icb iasplt* chi bpliiion •mciaMt b ititcacnti 

dUUias <vicB luimao'riiieurcc d«veJLopoftnc. ri««i« r«ipcpo4 Co ••eh •c^tc»«at siviag •• Accuracy a r«praMnui« 
cioo •! sroor bsliaf^ ai ia pciii&Xi* 



it to UAS EACH STATOCNT CASZTULtT ARD 

or THE Boxp lim to each staiehuit. 

'I 





/ X 




7 3 


/ 4 




HI fcacabXiihint aa acmosph^ttoC eooforc and Cruic 
im cmciai-tn any kind of cxccodad incarparaooal 

-problia iolvxht inEeraCCi©n. 










Pi 


C2l Parapbraaing or«rciCacins vtiac ahbtbar parioii 

kaa aaid: ia cha*baic vay co aoeouraca cha bchar 
»atribh to cohEinua ealtLittff- 












lh*V»^»? »P9)f?*o.c fbaX of probl«« bolviail^i^ 
tkat tbb pcrioa vieh cbi^* problen aekaovledga it 
— ai hi i /hag own. 






■ 




■ & 


CI) £fiftccira uia ot problar. solvms tlciIXt s^^Attf 

axda yoar af fores co naka en cha job coocaec 
vieh pabp^la -obra pro^ucctva. 








^' - 




^) Halpiat p«opU Co devalop ikiUa can only^^C^ 

ftaec ivbes bocb i^opla. involyftd baliava chac cha 
ooa baihg coaehid it abta and tfiXling co dtvalop 

^tha- akl 1 1 a f in -Qoeaci on. - 












ftaapacc i*. cha coonon danotaxnacor of all 
affa^ciVc hclDcr bchaviori. 












(7^/ Spa^ialVakiXti afa raquircd co 

^aiti^^Xy infXuahea an incerietion 

vfai^ ia cenCcrad on a probXca* 

1- , 1- ' . 












t§} GAOf audic acaff oparaca__ac a bisb lavaX ni 
ilitfarpefidnal cdmpacency. 












C$) A training couria on inccrperibnal eoQnunication 

ni iacarpa?«onaX probXcm ioXvinc onXy scCi in Cha 
- _yiy-bf Che •'real" work of CAO. 












ClOl A^ pa;rion bind by CAO in ah audlC position 

p<^aacasci \chi naeciiarr charaeCcritcict Co ba — 
able CS cooauhxctce affaccivelr- 












C^ll) X aa curxaocly very avar«-TSf how atillful X as ae 
farforaanca counieling (conyariinfyith ochera 
■bout thcix\ faaXinss -or pacioaaX nacccra ai thay 
Talace to cKe-per«&fl^a— pe^^at^ince). 












XUTwy pcrabnaX l»kiUi 4C pttrformaDca counaaling 

ata highly ^evelooed^ ' • 












C13) I Mm currcncly very avara of how ■kiXlfuX I aa 

at interpersbnal cotonunicaCionT 












Cl«) Hf^ periooal Ikilli ac interpersonal conssunieacion 












. Ct5J I fael very confident cha C I can be luccaiiful at 
a^j^Xyini apptopriace tkilXi whan engaged i6 p^r-» 
' foTMsee cbunielini^ and ihcer^erionaX cotamunicaeicm. 













(9) 

J- 

(to) 

(12) 

(13) 

(14) 

(X5) 

(16} 
(17) 
(18) 

(19) 

(20) 
. (21) 
(22) 

(23) 
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PRECOURSE 



DIFORKATION ASSESSMENT 



Zoatructionn ^ 



St 



This A39e9raeQC i» designed to measure a person's undiri 
aSoiic interpersonal conasonication akilia prior to participating in thia 
couraes. At the conclusion » a simitar assessment will be performed to 
decertai^ how effective the bourse vaa at increaaing the participanta' 
kaovledge and awareness of interpersonai skills. 
• 

I < This aiseasment has 2 sections each^ith a number of items. Read 
ii^tbh itetnr carefully, appl^ vfaatever inforration is appropriate, and aelect 
thi respdnae vhich best completes or anawera the queation. Check the 
-box in front of th$ response yon select. 

^ Hake sure you answer every itcmi \^ 

I _ __ ^ __ 

_ Juat before you begin to take the assessment, record the time under 
Time Started. When you have completed it, record , the time under Time Scopped . 



Tine Started: 



time Stopped: 



(24-25) 
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(Excerpts) - . mcooitSE 

WbtKATIoi A$SC9S)G1IT 

Zii»eihm eoimiaicicioa 4aMg GAO staff cm ool/ oeeor vtuin iadiv£4u«li iavbi^<i<l irt viit&R^is 
mi M% to: (^€ctt oai) (26> 

.■ : 

^* * ^ ^ Cnm<nic»C€ at^dt. 

3« / / tdunttfy ftldtiti for p^crfotsiaec oabsactMOE. 
4* / y iftsolvo ittCsrforinK vock r«l«C84 probloM. 
S« All of cHo ibdvi. 

Xt ia iiq^ortnc to (UO chiic oxplor^so b4rro offiettsi faolptaf itclUi boeooaii tluiso skills: \(Chtek 

ooo) ^ : Wi 

^* 1 ^ £ap«ct on tho nsatol boolth of tbo org^lutloo; 

^* Pmridc a stiiuUciag wbA iaecMrsglni ixporlsnes fot-Macgors. 

_; _. _ _ _____ _ e 

3* Ooeroiis ths ll^tOxbod that aiaagars will Croats furchor iatarporsoaal problsna. 

4* rn Ci) and (3) ooly. 

5* and (3K . 

til of t&o follbviag ari raaa9n:< tbo gcoeric loodel for intarperscmal holp kas special valua for 

^^i^oxr^tS'amM axe cp c s (Cheek euej (28) 



1« ^- -/ It jprorides a guide for tba helpxxig process. 

2* f J It prevents fmstratioo of the process. 

- / ^ -_ _ . ' -_ _ _ _ ■ ^ 

3« / / It aasUts the nahagcr to project his/her own needs into the process. 

4« I 1 It proridea a dcfinicioa of core ekills needed ia^the helping process. 

/ / Woae of Che abora. 

All of the fbllcMHag are effective Icinda of helper responaea exeevr: ' (Check ooe> (29) 

\m f 1 Aakiag "why" ^tueitiona. 

3* / / foet&eini^ oa aapects'lif the problcai. 

J. / / Focusing bo ciharactcriatica of the p«?traoo. , 

4* fZJ (13> «d (3). 

S. / / looar of the above. 

Vhich •aoager behavior ytmld^bei^ acc^H enplofee centered^ 

ott'a ^obicait ao chat Che ettployee has che ex^triencc of bei&g valued? (Cheek one). (30-) 

: £U Telling Che eiiployce betvecn phone calla chat he/she thouia get righc Co che point. 

_- 1^ • • . " 

^* f 1 Telling the esployee co eoiie inco eha office and Chen clociag the door. . 

^* 7 / "*'3liccxx>g on eomfo^s^le cbaira on oppoaice aidea of cha roo« fron one anocher, 

4. / / Talkibg vich/^hi employee item behind a deak. 

Sw' f I Sicciag oo a eooch righc ncxc co che eBptoyee. 



1 
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1 

Obstrvacion Zxerciil€ 



' - - - , \ 

You are about to participite in an observation exercise* Ttifi purpose 
of this exercise is to deteradue hcv skillful jou arc at identifying certain 
types of comunication behavior in other GAO audit staffs which foil will 
vitness interacting on the TV monitor in a mbmeht. Both the content and the 
nature of this interaction are fictional. Bovevert it does represent a 
possible discussion that might occur betveen audit team members. 

• %^ - - - _ 

On the next page you will first find a: list of communication behaviors 
and their definitions. When you turn this pagts^ carefully read over this 
list with their definitions i After this^ you will observe ah interaction 
of an audit team and indicate i^ich of these conaounication behaviors you 
witnei^sed the team display* 

The interaction will be replayed and stopped at 30 second intervals. 
At th<r end of each 30 second interval » you will scan the list of cbsssuhication 
behaviors on the Observation Sheet and indicate their^presehce or absence for 
each interval indicatedi If you obser/ed the presence of any awuht of a 
behavvor in an interval, yp^ '^^P^y P^s^e Che number one il^ in the box oh the 
Obsewation Sheet that corresponds to the behavior bbser^ not 
6bsen;e any occurrence for f pa^ticuiar behavior in that ^^^^^^^i P^^ce the 
nvii&ber zero (0) in the box on the Observation Sheet that corresponds to the 
appropriate behavior. For each observation interval every conmxunication 
bet^avior needs have either a (1) or (0) in the appropriate box on the 
Observation Sheet. 

If you have any questions about the procedure, ask the instructor to 
clarify them for you before you begin the exercise. 



\ 

/ til I ID eODE 
1-4 

/ / / DiViSiON/OFFICE/REGiON 
5-6 

GRADE CODE 

7 

£_2-/ Card 2 

8 
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9 



pgriHittows or co?cawicATioK BEiixvioRs 



49^ 



2. Cacsrias- 
4. JMs£nt 



5* Fcobinx- 



-Tb«tc -Cfite ftuacs or nods ot caajnUatioo. Thmf include sixpU verbal 
Wihpvior»_likc_"uh huh-'i -rithc*'^ "x**-* and others^ at well as a 

fQBiKiy nod or bovcmcqc of the h«sd; 



-IdaBCifyios on^ cooptncinc oo Cha scrcnccha of ct&octaar. ftcforcactag of 
Bpoclfic aebicvnenci li alio ippropriaio* 



-SuMarUinc vfaat anochtr haa aatd. or raqueicins ieUtiftcacioa. Atao, 
tcatint viiAC you thivk you heard froa anocher. 



■toy rtipofisc tHmt e^Id ptroape the ocKcr pcrs^^Co call Kor/biaic If Into 
'"f'tbar «iucicion, aad c^^ Che dcvtlojMiiC of chc helpint 

procaaa; (i.a., ""Tou're ioadaquaca"; '*Baaleally you're; Oot^ veil**). 



■^P«"««^9^_*n^ foc«M4_<;«««ci<^ elicit nora injUorpacioo on pra*-_ 

▼ioualy BCDClonad siccriai^ or rkqudiclog 4knoebar Co axplbra a particular 



-T^ aecurace ideoicif ieacion of fnbtEor'a fcillfi|a and/br tha c^ the 
MBaaga and cha raflaccibn of fceliosa and coataat. Refcrt to tht explicit 
Bcaaa^t acat. 



0 * SpC bbaarvad 



Bdavior 


I 


2 


3 


4 


5 


6 


7 


8 


9 


10 


It 


12 


Ackaovladsinc _ ' _ _ _ 


























gtwtaging 




















































Ju'dsimg ! 


























Frbbiac 


























Bsflcccliis V 



























<9-20) 
(21-32) 
(33-44^ 
(A5-56) 

(69-80) 




50 



ATTACHMENT, i 



SPCD FCLLOW-UP QUESTIONNAIRE 



\ 
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IDEOTIFICATION INFOFMATICN 



Enter the identification infooiation requested belcw. 



1. Idchtif ica ticn ^^jde 

Enter the four digit idOTtifiiMtion c^^ 

fcrar digits of yoor hoie telq^hone msiter) that yoa 

Mre t^f^ to use during treiining* If ycu did not 

your i*3n«^^§i^'>o??^_ nii^^ 
ber your suixstitute cxx3e^__please lobk to see if you 
put it soteuhere in your SPCD jraterieds. If you 
cannot locate or reczill the identification dcxie ybu 
used« please enter 9999. 



2* Region/Division Ocae 

Enter, your region or division code, 
listed below.) 

tieaKSguarters 



(They are 



01 


Oonnonity and Eooncmic Develop- 


13 


Atlanta 




liif^t Division 


14 


BOG ton 


02 


Energy and Minerals Division 


15 


Chicago 


03 


Federal Personnel and Ccnpensation 


16 


Cincinnati 




Division _. 


17 


Dallas 


04 


Field Opextitlons Division (HQ) 


18 


Denver 


05 


Financial and General Meunage- 


19 


Detroit 




ment Studies Division 


20 


Kansas city 


06 


General Government Division 


21 


Los Angeles 


07 


Hiiian Resources Division 


22 


New-Yoric 


08 


international Division 


23 


NorfdUc 


09 


Logistics and Ccmrunications 


24 


EWladelphia 




Division. _ _ _ _ _ 


25 


San Francisco 


10 


Office of the General Counsel 


26 


Seattle 


U 


ProcureCTient and Systems Acquisition 


27 


Waisliihgton 




Division- ^ 


28 


i*rari)cfort 


12 


Program Analysis Division 


29 


Honolulu 



30 Panama city 

31 Ot^ii^ (please 
specify) 



Enter the nunt)er of months, that have elapsed 
since yoii attended the SPCD bourse. If ybu 
have not taken the course, enter the nuniser 
**88." 

4. Grade^-L^el 

Enter your current grade level . 

5. Job Function 

Snter your 30b function code. (They are 
listed below. ) 



/ / / 
9-10 



11-12 





Job Function 


Series 




iJcb FSSiction 


Series 


01 


Adjudicator 


950 


lb 


Management Auditor 


343 


02 


Aiiivinistrative Officer 


341. 


11 


Mathenatical Statistician 


1529 


03 


Aiiiitor 


510 


12 


Operations Research Analyst 


1515 


04 


CdTpiit ^ientist 


334 


13 


Soiipr Analyst 


101 


05 


Data T: _ri±)er 


356 


14 


Socied Scientist 


i07 


06 


Editor 


1082 


15 


Statistician^ 


1530 


07 


Industry Eocranist 


110 


16 


Supervisory Program Analyst 


345 


06 


Management Analyst 


343 


17 


Supervisory Statistician 


1530 


09 


Manag^tient Assistant 


344 


18 


Other (specify) 
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DEFINITIONS OF ORGANIZATIONAL SETTINGS 



Individuals occupy this setting whenever two or more persons of an audit 
turn come IhCd face-Cd^faee cdhtacC with one another Co deal with organi- 
sational, idinihistrative, -or technical aspects of ah audit. The scope of 

Such neetings could l^e varied in natu_re_and could range from negotiating 

team agreements and planning to reviewing and discussing audit inforpiation 
collected on a day-to-day basis. 

Agency meeting 

Individuals occupy this setting whenever one or laore persons from an audit 

team come into face-to-f ace contact with individuals from' the agency or 

area on which the audit is-being performed.. Illustrations of. such meetings 
Would include entrance conferences, agency interviews, and exit conferences. 

Personsl perfortnance and development sessldiv 

^"^iyi*^"^^" occupy this setting wheneve comes into face- 

to-face contact with a supervisor and obtains feedback and/or infdrmacidri 
regarding hisy_her^jdb pcrfdrmahce dr his/her personal development within 
the agency. Meetings dealing with performance appraisal , coaching, and 
?9y"f^.^^"8» those focusing on career, planning would qualify as 

meetings which would be appropriate for this cacegdry. 

Staff meeting 

Individuals dccupy this setting whenever three dr more individuals from a 
given drganizatidnal unit (eig. regidhs or divisions) come into face-to^ 
^^*^^^9"^^"^^ y^th one another to_deai with issues, tasks, or problems direct 
lY_?'?lev^nt to that orgainizat ional unit . .Large . annjua I dr quarterly staff 
meetings invdlving praccicatly all inaividuals in the unit td more limited 
dhea such as EEO staff meetings wduld be . apprdprlate for this category. 

Personal problem solving session 

^'^'^"Py .^^^^"^ *_®tting whenever one indiyidu comes into face-to- 
f?9?_co"tact with another and receives sdvice or information on how. to 
better handle personal problems related Cd such themes aj alcdhdl, drugs, 
health, family, etc. 



DEFINITIONS OF CO^^^^J^^ ICATIQN-BE HAVIORS 



Using Bddy Atteritidh : The use of one's body to convey that one is inter- 

*^9D9*^Ptrating on what is be' - jaid, such as leaning slightly 
toward the individual or establishing eye contact. 

j^ ing C cmii^ . Those behaviors which convey sincerity, that the perSdh is 
being hxs dr her "real" ceif and chat helping ix hdt a special rdle. 

Inviting . Thof verbal responses which request another to begin or continue. 

Acknowledging . Grunt. and nod CdirnnCihicatidh responses like "yes", "uh huh", 
• ^^^^ tell the individual attention is being paid to what is 

being said. 

Ref leetihg . Feeding back Cd another the feelings and/or the content of s 
message. 

Sel f-Disclosi ng . Briefly sharing s similar experience from one's own life 
td encdurage further cxploraf'on of the prdblera. 



PgriNlTIOWS OF COMKtJfWiCATION BEHAVIORS (continued) 



7. U»inR Immediacy . Pointing but^*^here and n^^ to il- 

luitfate a point; to explore the nature of arelatipnahipi . or fo let another 
fchow how you're feeling. Example: "I'm ieeling i bit cotiiuiL right now." 
or **I don't know what quest iohi I should be asking you right how". 



8. Frobthg . Asking open-ended and focused questions which elicit more infor- 
aatxbn or request further exploracion of a particular area. 

9; Checking/Clarifying . Use of'the short restatements of whit the other has 
said (1) to be sure that the message has been undericood snd interpreted 
correctly, and (2) to maintiin contact,. 

10. Problem Stating . Developing stateuenta which accurately describe some con- 
dition that is to be changed. 

. .. _- ^ 

11. Action Planning . Collaboration with another to develop action plans. Action 
plans may include training, practice, homework, contracts^ etc. 

12. S u m marij c^ng . Re^ capping or listing the major points and cohcluaibhs co- 
vered during your conversation. In a sense, sunmia rizing f («n_extended 

form of checking/clarifying because the. sunmarizer wants confirmation that all 
concerned have the Mame understanding of what has been said. 

13. Ccntering/Reinf orcernent . Helping the other person to recognize hie/her 
strengths, - to focus on him/herself.and on the qualities that will enable 
that individual to take problem iolving actions. 

14. Appreciating . Acknowledging another's strengths and the value of these 
strengths in the conversation. 

15. Judging . Any response that cpuld prompt the other person to call hio/her- 

self into further question. _ Example! "Are you sure that's really the 

problem? 1 know your team leader and she doesn't usually lose her temper 
without good reason." 

16. pi^otHit^ng . Any response that implies the person is making too much of a 
problem. Example: ''it's not all that bad. . ." 

17. Ego Speaking . Any response that shifts the focus_to you and your experi- 
ences instead of the other person's. Example: "I know just whfft you mean. 
That guy makes me angry too. Just last week. . ." 

18. Being Irrelevant . Any response that ignores the problem usually by changing 
the subject. Example: "Yeah, well, I sure ain glad it's Friday. Sounds 
like we could all use a weekend right about now." 

19. Being Pollyannish . Any response that encourages optimism without offering 
any grounds for Example: "it's tbu|h you lost your leg, but at least^^ 
you have anothrr. Don't worry. I'm sure it'll all come out for the best. 

20. Being Sympathetic . Any response that , topugh yellrweantj^ offers t^.^ other 
person ueither (a) your assistance in stating the problem in solvable terms ^ 
nor concrete suggestions of other avenues to pursue. Example: 

"I'm so sorry. That's just terrible. You really don't deservi such a rough 
time." 

21. Dump log . Any response that either (a) implies that discussing the problem 
or doing anything about it might make things worse; or (b) suggests that 
the problem is already much worse Chan the person feared. Example: "Man 
are you in trouble. Once he loses confidence in you, it's all over." or 
"If you go around saying things like that, you'll really be in trouble." 
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inst ructions 

Five organizationit ieC-Cingi are. tilted betov. (Their def ii\itidhfi are liited 
on page 2.) Refer to these •ettlngt as you conplete tecnlont A, B, and C of 
the questionnaire. 

Section A 

Place, a iJ) check in the box v^ich repreaents the n\ffiber_of .ti»ei that you 
have been present In the organizational setting since attending. the.SPCD 
Workshop. If you have never been in 1 parti ciilir iettihg, check (w^ the 
^P'^ "D?^^.*^ to the next setti^ have riot attended the 

SPCb course, report the number of _ times that you have been present in the 
Dtgahizational setting in the lost six months. 

Se^t^n-& 

jtefer to the def init:ion8 .of eoamunicatidn behaviors . and their codes listed. . 
th pages 2-3. Enter the codes for up to four behaviors which -you can recall 
using with some frequency for each organizational setting. 

Section C 

Place a if) in the box which represents the extent to t^hich you .believe_your 
interpersonal effectiveness hcs changed s ince attending the SPCD Workshop... If 
you have hoc attended the workshop, place a C>f) check in the box labeled "not 
applicable." 



— OftGAHIZAtlDNAt SET TINGS 



Huiaber of meet- 
ings attended. 

(Check one) 



B 

Communication 
behaviors used 
by self. 



ypur _ interpersonal 
effectiveness . 



Enter up to 
four codei. 



5* 



s 



1) Audit team meeting 



2) Agency meeting 



3) Personal performance 
and di^velgpt ^ent ae ssitm-. 



4) Staff meeting 



5) Personal problem sol- 

— ^vin g s es sion. - - - 
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PXRT II 



Place a (/) check in the box vhich b«tt representB your feelings regarding 

each of the questions below. If you have.aot. attended the workshop, place 

a check in the boxes labeled "not applicable." 

• - • * 

I* $9 wh*t extent has the caused you to be more aware of how /63/ 
you behave when you cotasiunicate? 

f— J Little or none / / Some / / Moderate / / Subitaatial 
/ / Very great . / / Not applicable 

2. To what extent has the SPCD course caused you Co be more aware of how IJbkl 
others behave when you communicate? 

/ / LjlCCIc or none / / Some / / Moderate / / Substantial 
/ / Very great 7 — / Not applicable 

3. To what extent have you found the material taught in the course to be /65 / 
of benefit to you when you cbasnunicaCe? 

/ / Little or none / / Some / / Moderate / / Substantial 
i — / Very great /— / Not applicable 



PART III 



Instr^ 



Consider that each of the statements in Part III has been made to you at the : 
beginning of ah interview. Following - each statement are a number of alter- 
native F?*P°0?§^i Place a check in the box cbrrespohdihg to the alter- 
native that demonstrates the greatest degree of regfieet (i.e., regard, esteem 
or courtesy) itl the situation. 



"I'm really getting fed un with th£Se blasted reports. It looks like every- /66 / 
thing has to be done yetterday. Why in hell can 't we get a little_mbre 
notice. That *s no way to do a good job. What's so rough is that I can't 
count on_most_of my people to understand exactly what's required and not 
spend a lot of time getting information that nobody is really asking for." 

a. I 7 "I_b*ye_ the same feelings. But that^s ju8t the way the 

government. works . Somebody at the _ top wants_some silly in- 
formation Chat- they won't know what Xo do with after they 
get it. Probably they weren't all Chat interested in the 
first place. But by the time the request gecs down to us, 
you'd think the future of the country depended on it!" 

b. i I "To hell with it. You've got to learn to quit worrying 

about quality. Quantity! Thatjs what people are inter- 
ested in. Give them something big and fat and they'll 
tell you that you did a great job." 

. c. / 7 "lt*9 hard to keep interested in doing a good job when you 
fe^l you have to fight the whole system to do it." 

d. / 7 "I guess the name of the game is to quit worrying about 
T what's -important and what isn't and to do what the boss 
says." 
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TAUT III (continued) n . 

^ _ _ . _ ^ _ 

2» "Next month I have to ret ^ doeen' t m«ke_ ienie. I^^B^ but 

^|_york M lot herder than e lot of theae younger people around here." 
^Check one) 

a. 7 — / "I've a Iway a looked fqryard to retirement myaelf . I can't 
* vjit to get on that boat of mine." 

b. / / "It* a tough to know you've at ill got What it takea to do a 

S9P^ 29^ and the company aeema ro rule you out becauae of 
your age." 

c. / / "Good grief! You ahbuld feel lucky to a till have your health 

and the chance to do what you want for a change.*' 



d. / / "How doea your wife feet about all thia?'* 

e. jIZJ (b). 

3. "I don't think I*ve ever felt so humilated. The boia* aecretary called and 
told me he wanted to see me in the conference room right away. When I got 
there, he eaya, *Where*8 the Appleton. report? * I aaid, 'What, report? ' No- • 
body told me- to prepare a report: But What really Hurti Is that riobddy 
believed me." (Check one) 

a. / / **Maybe_ you were so upact that it just aecmed that way- I 

caTi*t believe lidbddy believed you.** 

b. / / **Sounds like the worst part of It is having your boss feel 

you can*t be trusted.'* 

c- / "Hah you are in trouble. The way thAt boas of yours works. . . 

vhen he besina to lose confidence in you it's all over.** 

d. / 7 **Sdunds like an organiratiotial problem in dyafunccional com-* 

muhicatibn networks.** 

e. / / None of the above show respect. 

A. '*The promotion practices in CAO really stink. If you could make it on merit /^^ / 
I sure as the devil would have been on the list today.*' (Check one) 

a. / / **Blaming your failure on the system isn*t going to get you 

anywhere . ** 

b. / / "Iknowhow much you were counting on your promotion, _ but 

things have a way of tiirnxng out for the beat in the Idtig run." 

c. / / **I really feel sorry for you, Harry. _ You and I both know how 

much you c served to get promoted. It*B really a shame.'* 

d. i — / "You feci you have been handed a raw deal and you* re really 

disappointed about it." 

e. /~7 (b) and (c) 

5. '*I*m really disappointed. I dldn*t.get permission to attend that course on /70/ 
systema managewent next week. I think I deserve better treatment than that. 
Vhat*s the good of breaking your back to do a good job, if no one is going 
to help you improve yourself ao you can get ahead.** (Check one) 

a. / — / "I'm really sorry. I feel as bad as you do, but what good 

is that?" 

b. / / **It hurts when you miss a chance tO improve yourself — espe- 

cially when you know you*ve been working so hard to .get 
ahead.'* 

c. 7 — / "They'll surely give you a chance later. YOur turn will come 

later. Juat wait and see." 

d. / / "How many others from your office received permission to go?*' 

e. / 7 "I know, iota of people wiib were turned dOFwri for that course, 

id don't feel ao b«d;" 
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INTERNAL REVENUE SERVICE 

I 

LEVEL II EVALUATION . 
SUMMARY 

On-job evaluation of training at IRS is called "Level II" feedbaclc or external 
evaluation. It Involves collecting course criterion test scores and 
background information on the trainees and mailing out iquestibnnaires to 
trainees, supervisors, and /of on-the— job coaches iseveral months after 
training. Information is collected on the frequency with which tasks are 
performied, the adequacy of training, in preparing participants to perform 
tasks, and how well the tasks are carried out. The subsequent data is - 
processed, stored, and displayed by means of a computer program, and the 
results are used to document and improve the relationship >??tween training and 
job performance. , 



i ^ .; 

i; 
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G^ERAL DESCRIPTION OF THE EVALIJATIC^ METHOD 



Level II feedback Is desl^jned for use with technical trainin^j courses at 
IRS. As an exatnple^ the Revenue Agent, Unit I (RA) training courise evaluation 
process is referenced in the following review. The basic methodology, is the 
same for all courses, yet aspects of it vary depending on specific 
characteristics like the student population, training program design, and the 
statistical analysis required. 

RA training was a priority program for evaluation because of ithe large volume; 
of trainees, ranging from 500 to 2,000 per year, and because RAs are 
considered among the top professionals at IRS, They mast have accounting 
degreeis or the equivalent to be considered for employment, and ©nee employed, 
they participate in a four-phase training program. This evaluation was geared 
ta.unit one of the training program, involving eight weeks of classroom 

instruction and a subsequent seven weeks of on-the-jobf training (OJT) 

y 

requiring application of the tax laws and examination techniques in which they 
have been instructed i ^ 

A prerequisite step in developing the e. iluatlczi methodology involves 
producing the list of job tasks on which rhe courGO is based ^ the subsequent 
course objectives, and the course criterion test, T^ei relationship between 
the task, its objectiveCs) , and its represen-.: ative ? ^.st: Itet/ is imperative 
order to measure how wei 1 the objectlv.^i aie i?.^ ' . ' 1 -^s cc^; : sd to how well 
the task is performed on the job. After thi^a --r...:' r ' is rjc. ablished , the 
following steps make_ap the evaluation proc^«fr. 



K- ■ ^ 

f ' ^ • / ' ■ ' • ' ■ - 

I Statement-Q^^v^Iuat ion quest tons- - There are two questions always addressed: 

\ i. Are the students who successfully completed the course able to 
• perform the Job tasks for which they were trained? 
2. Are the tasks selected for training appropriate to job needs? 

Other questions are posed by the person or persons requesting the evaluation^ 
that is, the clients The client may be interested in attitude changes * 
regional differences, etc., and questions are developed to reflect these 
interests.. ; . ^ 

Selec£i^- of— evaluation instruments - The, evaluator decides which instrument js: 
are appropriate considering the objectives of the evaluation and the ' 
characteristics of the course. Tests, questionnaires, observations, 
interviews, or surveys may be used. The IRS standard system uses class 
criterion tests, questionnaires, and a cdmput^ data analysis program (using 
the PLATO instructional computer system). , . The sources considered for data 
collection are students, on-job coaches, and/or supervisors. 

Selection of sample and preparation of action -plan - The evaluator decides how 
to ope rationalize the revaluation process. A sample of classes is taken. In 
RA Unit I, for instance, eight intact classes (25 trainees each) were selected 
representing different regions of the country, different district sizes, and 
different recruiting times (taking into account the variables deemed 
significant by the client). y 
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In RA TTnlt student test data is routinely entered into a cbthtJuter test 
program* Test data for the saratDle was dumped into the Level II "^cbtnputer data 
analysis proji^ram at the termination of the courisie. Studentis and otl-the-ldb 
coaches were notified of the nature of their future input'at this time. 
Anonymity of trainees* scores and responses was thus assured.. . ^ 

A questionnaire measuring job performance was mailed dilt at the end of on-the- 
job training, to each: trainee and his/her on-the-job coach; (Note: For most 
training courses, it is best to mail the questionnaire to each trainee 
and »uper?/isor three to six months after coarse completion. However, RA Unit 
I is followed by seven weeks of OJT and then another seven week course, RA r 
Unit II. As a result, the best job performance data for R^ I is obtained at 
the end of OJT.) w - 

DesigyLof-instrumenLs - The evaiuator designs or ada^^ the instruments 
selected and pre-tests tnem. For instance, RA trMnlng employs criterion 
referenced tests and , •£ oi;. the Level II evaluation, a questionnaire mailed 
several months after the completion of training. .With criterion 'referenced 
tests. Instructors evaluate trainees on how they perform in reference to each 
course objective. The questionnaire was used to accumulate backjB^rbuhd data, 
task ratingsi^ and j^n^ral responses. (See Attachment for questionnaire 
instructions Brd content.) The background data In the questionnaire allowed 
for multiple-choice responses to items mainly inquiring into academic and 
professional backgrounds of trainees. ^ 

The task rating scale involved the evaluation^by the trainees and on-job 
instructor of each task performed on the job after the termination of 
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course* Each task was rated on three dimensions: frequency, training 
; adequacy, and task performance^ In reference to hew frequently a task is 
performed on the .Ipb, the evalu^tor could see If training was relevant to what 
the trainees actually do at work. For example, in RA Unit 1 evr'uation, the 
task requirlnj^ trainees to locate and cite revenue rulings and court' cases was 
rate*^ as: a) never performed, b) perfbrmed for one or two cases, c) performed 
for about half of the cases, d) performed for most of the cases ^ or 
e) performed for all of the cases* 

The second task rating involved the adequacy of training as input for future 
training course offerings* Usin^ the same taisk as an exaiiiple, trainees and 
bn-the-^1ob coaches rated whethier: a) the task requires much more iemphasis in 
training, b) training less than adequate, increase emphasis, c) training 
adequate for task, d) training more than adequate, decrease emphasis , or 
e) greatly reduce or eliminate training for task. 

The last task rating involved how well the task was ^performed on the lob after 
training; A^ain in reference to the task requiring trainees to locate and 
cite revenue rulings and court cases, trainees and , on-the-job coaches checked: 
- a) could not perform task, b) substantial errors in performance, c) few errors 
in performance, d) satisfactory performance , or e) superior performance* The 
analyses of these task- ratings were' a large part of Level II feedback* 

'^he general question section was reserved for any additional questions not 
addressed in the task ratings, e*g.", '*Were there any job tasks not trained 
which should have been?" 
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Collec ion of course criterion test resuits - Subjects' test scores were 
collected by their ID numbers frbin the computer program and a mailinj? list for 
the distribution of the questionnaire was created. 

Admin istration of instruments ^ The evaluator administers the instruments bv 

^ _ _ 

means of a'mailing or by other means appropriate to the circumstances. If ^ 

mailed^ the instruments are returned in an enclosed self-addressed mailing 
envelope, 

\ 

■ ^ 

An alysis and int^er^etatlor^ of^ data - All responses to the Level II 
questionnaire were used as input to the computer data analysis program. The 
evaluator examined the dsta produced for information to improve the course. 
The background information came into play as a population selector. All 
responses can be analyzed thrcrugh gx^ouplngs of the background ci. ^:a, and 
various reports are generated in terms of each category of trainee: college 
graduate, non-college graduate, number of years of job experience ^ GS rating, 
region » etr« Reports of individual groups, combination of groups, and the 
toC;^l population can trt tade: 

!• Means an^ stanr. ri deviations of zW task ratings (freqv*erc>% 

1 1 : in ad eq u c y , \a rd j oh pe r f o r ma nc e ) o 
2i Fv 1 y and percentages of all task rarinj^f - 

■ ■ * 

3, Zo^rsc -erion test scores by class ^ 

Performance by objective, tha is, the percentage of traiizee^, who 

Eiastered sp^^cific objectives. 
5, The reJatlbnGhip of performance the criterion tCst to tasl:: ratings, 

thereby showir):^ how the masters or non-Tiia^tory of an objective relates* 



to iob performance on that same task. An example follows: 

/ ^ 



. Task Task Training Task 

Obfjective Frequency Adequacy Performance 



/ 



Obj Met? N Mean SD Mean SD Mean SD 

I 

Yes 24 ' 2.50 0.82 3.1^0 0.65 3.00 0.^6 

001 No 0 

002 Yes 20 3.00 0.82 3/.65 0.87 3.50 0.65 
No 4 3.10 1.00 3.02 1.00 3.00 1.00 



/ 

! / 

.j. . . . . . . . 

In ob^1ecL\vje 002, those who passed thja training ob.lective also did better In 
performing the task dn-the-iob. Thisj report can indicate which oblectives are 
crucial to be mastered in the trainijig, in order to assure satisfactory job 
performance. | 

/ ■ 



6. The analysis! of the general questions permitting quantifiable responses 
by frequencies, percentages, meians, and standard deviationsi 



7. Correlations of any two Items,' e\,g., the relationship between grade 

, ■\ ^ / 

^ level and task performances \^ 

_ _ ; \ ; . . 

inl the past, the questionnaire responses fi^^om trainees and from supervisors or 
OJT coaches have high^ly correlated. If a d-Lfference does appear, the 
trainees' answers on pask frequency and tral\^lng adequacy are given more 
weight, as they have the most direct knowledgii about ^£^at_ is occurring and the 
relationship of training to the performance of\:1ob tasks. The supervisors* 
answers on the qualityl of task performance are given, raore weight ^ as they can 
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make a more experienced iud^craent on H o^ well the tasks are being done. 

Preparation of the f inal^rafiort - For Level H efforts, an evaluation report 
is prepared^ including a Summary, backgrrand information^ a description of the 
study i results and conclusions, and recommendatioT?^ i Results a^e primarily 
used for course revision. In addition, the eVaiuator can use the data to 
compare different instructional methodologies (e.??i, traditional classroom vis 
a vis computer based training approaches), by ? oo/.ing rt the Level II results 
for participants from each ^^roup. Using t :»e ; ^ickgro-^ data from Level II 
questionnaires^ the organization can also tra:k prc^^ress of entry-level 
professionals to determine what types of indi xduals stay witii 
organization and the levels of performauce they achieve. 
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ANALYSIS OF EVALUATION METHOD 



Conditions for usig 

- List of tasks, 

- Course objectives stated in behavioral terras and related to the job tasks. 

- Standard quest ionnaireis and iriventorieis. 

- Computer program (IRS uses a program desi>l;ned for the PLATO syiStem; 
however, other computer prdp^rants can perform similar operations); 

- Organizational climate which supports (expects) thoroo^^h evaluation 



Staff time is required to acquire data through the queistionnaire, to Input it 
into the computer program, and to interpret it. The design of the 
questionnaire, the interpretation of the data, and the final report are 
probably the most time-consuming phases and should be performed by someone 
familiar with t^ie program and with the analysis of non-complicated statistical 
data. Step-by^tep guidelines for developing, conducting^ and reporting the 
evaluation are being prepared during 1983 for IRS program managers. 

The input phase involves typing parameters of the specific course being 
evaluated into the computer data analysis system (number of objectives, number 
of trainees, etc.) and likewise the parameters of the questionnaire (number of 
background questions^ number of alternatives for each, number of tasks, 
etc.). The completed computer score sheets (used for questionnaire responses) 
are then run through an optical scanner. 



effort is. 



Re sour 
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the trainees' and supervisors' efforts to fill biit questionnairiss several 
months after course completion usually take about thirty-five to forty-five 
minutes per questionnaire. 

Nature of information produced 

The information provided by the data analysis reveals how the course affected 
job performance r.n.^ vhether the tasks trained were needed for the job. The 
course develc^^ " ov trainers can use the information to revise the 
instructional objectives and the design of the course. The reason the 
analysis is so effective goes bark to the basic organization of the course.' 
The course objectives are der. fr^^^n the job tasks, and test items on the 
final ex^^ measure achievement of t^^e ^bjcctivas. The final report of the 
information can also be used on the job by supervisors and coaches to assist 
employees with cheir fu;. ther developTTient . 

The population selectcr asr^ct of this methodology is important. For example, 
in a total group report, a task r y show a low performance rating; however, 
separate group reports created for trainees based on background variables such 
as experience, grade level, or location may indicate that a particular group 
is having problem in performance rather than the entire population of 
trainees, this will help tnclividualize training for specific populations. 

Ad^vafitages^ ? 

The p.moply of information which can be retrieved from background data 
input (such as regional and educational differences); from self- 
perceptions of jol5 performance, supervisor's perceptions, and their 
comparisons; from the relationship between tesj performance and task 
ratings vjrtthtn various populations; etc. 

• 7;. 

o 

ERIC 
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- Quantifiable rather than anecdotal data. 

- The different ways the data can be analyzed uslnj? the PLATO program. 

- The ease with vhlch decisions can be made. 

- Coot-effectiveness of mall-out questionnaires and proven reliability of 
data. 

- Anonymity of subjects and emphasis on group performancie rather than 
Individual performance, making the evaluation less threatening to trainees 
and unions. 

• • 1 

Llmltat Ions 

- Course must be based on job tasks and must have criterion tests. 

- Former trainees must have the opportunity to perform the tasks on the job 
within - few months after trainings 

- Requires q^icstionnalre response tlnxi from the trainees and the 
supervisors, whlt^V^ c^iti be long if tVe tasks are numerous. 

- Amount of data generated requires computer processing, particularly if the 
sample Is large. 

Refere^jce 
Marjorie Kupper 

Program Manager for Evaluation 
Methods, Media, and Services Branch 
Training and Development Division 

Internal Revenue Service ' . 

2221 Jefferson Davis Highway 
Crystal City, VA 22202 • 
(Phone: (703) 557-2456) 

■ . y.' 



ATTACHMENT 

EVALUATION OF PEV^^iNUE AGENT UNIT I 
CLASSROOM TRAINING 
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Evaluation of Revenue Agent Unit i Ciassroom Training 
—Directions for Trainees— 



Your Revenue Agent Unit i class has been selected as part of a special 
evaiuation study to be conducted by the .national office. The study is 
designed to measure the effectiveness of the training and to find ways to 
improve the training. It villhbt be_used to- measure thelndividual 
trainee in any way . Several classes from several regions are involved in 
the study, and all results will be expressed as group averages and 
correlations • 



The spfecific purposes of this type of evaluation are 1) to determine if 
trainees ;can perform the job after completing the classroom training, and 
2) to see if the right tasks were trained for the job. To accomplish 
these objectives » we must obtain data regarding the trainee during 
ciassroom training and at the completion of OJT. We have already 
received your classroom data; the enclosed, questionnaire will provide us 
with all information needed at the end of OJT, 



please fill out the enclosed questionnaire using the optical scanning 
Score Sheets for your rfespdtises. There are 2 sections to the 
questionnaire. The first section requires one score sheet; the second 
section^ requires 2 score sheets. 

Each section has its own directions. Please read them carefully. The 

first section asks questions regarding ybur_ background. The second 

section presents the tasks you learned in classroom training, and asks 
you to rate them according to the frequency you now perform them, how 
well you feel you were trained for them, and how^weii you initially 
performed then in OJT. This section also presents tasks you performed in 
OJT in association with your cases, and aslcs you to fate how well you 
performed th::ri, " 

Please be candid with your responses. This questionnaire can only be 
effective ±ti improving the course if we receive honest answers. Your 
social security number is required on the score slieets so we can track 
your ciassroom data to this OJT questionnaire. Alter the cdmputer 
matches the 2 sets of data^ all individual numbers and reports will be 
inaccessible. Our ^urposci as stated earlier, is to mea&ure training 
effectiveness, not the individual trainee. 

it is equally important tc '.he study t ha t^w.e g^ all 
trainees. _Please^return'air376Core sheets and the last page of the 
third section (requiring a written response) in the envelope provided by 
p^j^ 2:. Your on-the-job instructor is being asked to fill out a 

^iSllar questionnaire for each trainee so we will have 2 sources of 
information. 

7G 
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SECTIOl^: -BACKGROUND DATA 



Fill in the blanks in the following manner: 

Name : , Evaluation of RA I Training 
Course: Background Data Date 
Test Code: ^ Training Site 



Write your identification number (social security 
number) iti the first section of the score sheet. Write 
the number in the manner ishown by the example on the 
score sheeu* 



s 



INSTRUCTIONS : 

The questionis in this section are multiple choice. Record all youv 
answers on the score sheet. Give only^ on^ r^ponse- for- each question * 
If more than one response seems appropriate, pick the one that best 
describes your answer, e.g. the area where your major amount of work 
experience is, etc. As you answer, check to make sure that the question 
numbers correspond to the numbers on the score sheet. PLEASE ANSWER 
EVERY QUESTION . : 



Part 1 
(upper 

left 

corner) 
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1 . Are you/ completing thi& qu^s^ionnai^e as^ : 

a. trainee 

b. dtirthe-job r^nstructor 

2 i ' Hov yas your training conducted ? : 

a. Prototype class: self-instructich with 
computer-based training followed by 
traditional classroom presentations. 

b. All traditional classroom presentations. 

3. Do yoy_have-a^-4^yeax^colJLeg£ -degree ? 

a. . Yes ' ; 

b. No 

4 . How long has it been since you received the 4-year college 
degree ? > 

a. Not applifcable 

b. lesR thaii 6 months 

c. between <) months and i year 

d. between i and 3 years 

e. More than 3 years 

5. Identify your college degree major. 

a. Not applicable 

b. Business administration (including accounting) 

c. Liberal Arts 

d. Sciences 

e. Pre-iaw. 
a & e. Other 

6 • flow many credit hours of "pure" accounting do you have ? 

("Pure" accounting would include, introductory > ntermediate , 
advanced^ cost^ auditing, etc. It would not include finance, 
accounting for managers, business lav.^tax law, of any course 
that does not fdcus^^^^accounting' pMce 

^ ^ '""a, less than 12 hours 

b. 12-17 hours 

c. 18 ~ 24 hours 
. d. over 24 hours 

7 . Where did you get the majority of ywr a^e$iintiiig-r .rfedi£&? 

a. f6ur--year college 

b. junior college 

c. business school 
d; other 



7 • 



EKLC 



\ 1 

8. H ow m a ny cre d it-hours -Oj^tax-lav do yoa hav^ : 

a. None 

1-3 hours 

c. 4-6 hours 

d. more than 6 hours 

i 

9 . Do you hold a professional license? ? 

-- 1 ' 
. a- No_ \ ■ • : 

b. CP.Ai 

c. member of the bar 

d. other - 

10. Identify previous work experience in accountang> 
a. None _ _ 

b- government (federal* state or local) 

c. public accounting 

d. private industry - ^ 

e. other 

11. Hoy many years of previous work experience in aacouptdiig do -yog 
have? 



a. None 

b. less than 1 year 

c. 1-3 years - 

d. 4-6 years 

e. 7 or more years 



1^ 



12. I^ieftri f y the -Ocir ure^d^ f-^cuir— p^eviQus-work-&xperl&^ in -IRS . 

^- — a. No previous IRS experience 

b. Tax Auditor (completed TA I and TA II courses) 

c. Tax Auditor (completed TA III course) 

d. Taxpayer Service Representative or Specialist 

e. Revenue Off icier 

a S e. Accounting Aid 

b fi e. Co-op student 

c & e. Other 



13 . Hov was jyonr job interview conducted ? 

a. telephone interview outside your own district 

b. telephone interview within your own district 

c. office interview outside your own district 

d. office interview within your own district 
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14. I had an accurate view* of what the profession 
would require of me when I was hired.. 

a. Strongly disagree 

b. Disagree / 
cjlncertain 

d. Agree 

' e.. Strongly agree 



ERIC 



(Excerpts) 
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SECTION- B S^- T&SK-RATXNGS 



Fill in the blanks in the fpiiowing manner: 

Name : Evaluation of RA I training 
Goarse: task Ratings Date 
Test Code:. Task Training Site 



Part 1 

(Page 1 upper 
leit corner) 

Part-3^ 

(Page 2, upper 
left corner) 



Wr-'te your identification number (social security 
number) in part 1 on the first score sheet. 
Write the number in the manner sh bvm by the example 
on the score sheet. Do the same on the second score 
sheet in part 3. 



INSTRUCTIONS 

The tasks listed on the fbllbwirig pages are the ones that 
were used to design Revenue Agent Unit I classroom Training. 
Rate the tasks now according to your experience in on-the- 
job training. Rate each task according to three different 
characteristics. 

FREQUENCY - How often did you perform the task? 
ADEQUACY OF TRAIN . NG - How well did the classroom 

training alone prepare you for the task? 
JOB ■ >/^ORMANCE - How well did you initially perf otm 

Record your ratings on the score sheet with only ^me 

response for each characteristic under a task. As you 

record your ratings ^ check to make sure the numbers 

beside each task correspond to the nuabers on the score 

sheet. The Questionnaire atidxtht score sheet correspond to each 

other by number as sliown below Vith number 1 . 



QUESTIONNAIRE 



SCORESHEET 



4"ask 


Freguency^ 


Adequ. 


ac£ Perfc 










A. 




2. 


3. 






2. 
3. 
4. 
















5. 




B. 


4. 


5. 


6. 






6. 
7. 
















8. 




C. 


7. 


8. 


9. • 






9. 

10;- 
















11; 




D. * 


10. 


11 . 


12. 






12. 





RECORD ALL ANSlsfilS J 

eoHPDTER scGi: 1 


1. h'Hi r^-riiTivd 

b. Uthmi (or i or I ciirp 

c. Nriorwd lor ibmt ^ of tiiii 
i* ^n\omi ht Mit of uifi 

1. ^rlorifd for ill ciiii 


1 TMIMINC >^ACT 

6. tiik riquirii nch wri nfiHiiLi 
m triliiQi 

TriUlni Ifif.tUi iif^uitii 

Incriiii ftphaili 
c. triinini iii^iau (or tiik 
i* trilnlel Mri (liiii idt^Mtii 

4«criiii ttphiili 
It Griirh ruiBct or iliiitu'it 

trifnini (or cmk 


mt mmm 

1. Cfwid M( psrion till 

S Svbitinciii trrori {% p%rhmw 

c« FMirioriJn.^rroHiiei 

t. Sttf^rlor ptrfonifici 

Ht't niivtr ll iiik Itii 


TASf. . .. . _ . . _.l 








A. Lnrntp .ind die Coup and RpguKitlon firrtlnns. 



1^^ a b c d e 


2^ a b c" d e 


2 a J» c p 

a 


B. Lor .lit' .ind cite Revi'mie RullnRs *lnd court cases. 


4, abode 


5, a b c d e 


a b c d e 


i:. I)-cernlne '.U t^Hpayer's correct tax UabllltJ 
iisjrig flip ..ihles of tax rate schediilr^ 
wlilchever ,!*• i'.ppllcablc. 


7, a b c d ' e 


8; a b c d e 


9^ a b c d e 


i). Compute the prepavment credits for Earnni 
Ihcojiip and/' i Cxrt'<!S FfCA! 


,1G. a b c d e 

/ 


11. a b c d e 


12. ■» b c d e 


K. Dctermjiu' 1*^^ ' Trect deduction for perspiml 
exciiipUfJh'i; 


13, a b c d e 


14, a b c d e 


15 a b c d e 


F. Deter.tilne tbe correct fiiinR status for 
Individual income tax returns i 


16, a h c d e 


17.. a b c d e 


18. a b c ,d P 

1 


r. Identify Items uf |{ross IncoDe, and recognize 
certain statutory exclusions [rom gross income. 
Ruclj as gifts, inheritance, prizes, scholarships 
or felldwsliips; life Insiirahci? proceeds arid 
Interest on State and inunlclpnl nbllRatlnns. 


19. a b c d e 


29. a b c d e 


21. « b c d e 


!l fictcrnilne ulieflier an expehsr I" rlndnrf Ihln 
in tlie conduct nf a trade or business, ind 
(iist inf.iiish hetwrrn capital expenditures 
and operailnR exprnso*^. 


22, a b c d e 


23. a b c d e 


24. a b e d c. 

1 


I Deternlne the deductibility of travel and 
transportntion expense. 


25. a b c d 0 


26. a b' <^ J e 


27 a b c a e 


J Determine the deduct ibttUy hf criEcrialnnipnt 
and hjistriess gift ixpenses, IncludinR 
cniPttainiBont facility expenditures 


28. ^ ^ c d o 


29, a b c d 0 


30. a b c a e 



RECORD ALL ANSWERS ON 
COKFUTER SCORESHEET 

* 


tixlhm^ for l.et_l.c?'*» 
e. ffttm.f> hr%hm\Hr>t ccm 
d. firfon '1 ''.it »oit of t-jiw 

ftrforri'H tor /ill ««i 


*• Tiik mulrii wch ion tfphaili 

in triiilrj 
6. triliiii liii tluii t^i^uitii 

iKriMf tf^iiii 

c. TrilnlAi idr^Mtt for ttsk 

d. t/ilnin| "tort tiiM idquitt| 
' : . !l«srfm_ci|!biila 

1. Crntly ifdiKi or illilniti 
tralnini for tjik ; 


i m ftmma 

?• ??!!lfa?^P*rriri uik__ 
i. i^bitanUl irreri in ptrfaniRci 
e. !rv irroM In ptrfoHloncf 
i* fiittifictorr Hrfomnci. 
ii hfvivT ftrfominci 

(do w\ fntvffrJf .tiili hit 
?)tvfr pfrforifd) 


1 tASKS 








ff. Cofiputf thc inveatwent credit cflTryhnck 
and recapture. 


94, a b c d e 


95, a b c d ' e 


96. abed* 


1 — 

0/;, Cotnpute thf imhlt portion of an annuity 


97, a b c d c 


9^1. a b c d e 


^^9. a b c d e 


Ifll. DMfriiiliie the excluglohS ffol thcoliie. 
nllnwftblc undrr Code (ineala nnd ]od|>ing, 
or rental allowance furniahed to a 
minister, Insurance prenilum paid by 
; eiiipl oyer; death benpfJt; dflmogea^ workntan'a 
compcnftfltl';?, benefit?! from accident 
and hcaiti' H^ur/ince, dlflablUty incomr), 


196. a b c d e 


IGi. -1 b c J c 


102. « b c d « 


11^ Decennlne if a. loss, is aeductiDie> ana 
compute the allownblc canualty loss 
deduction. 


103. a b c d e 


— — . 

104. a I c a e 


105. ,i b c d e, 

i 


In connection with bualneiis use of a 
home and rental of vacation hoinpfl. 


106. a b c d e 


107. a b c f'^ e 


108; « b (i a e 


lUj, Diiiri j.'ilih between a Bualheijfl and a 
non- ii-lrf^' bad dabt, and determine 
'thf ,'i"<^tinf irtWflile flfi A liflfi debt 
dedu' • ' lag the. specific charge* 
off f.f, . -u'rve methods. ; 


109 a b c d e 


110 a b c d e 


1 11 » ► c d e 


IL, Compute Blnlffiuw \ < and the alternative 
ninim tax. 


112. b c d e 


113. a b c J e 


114. a b c d e 


KM, Coflioute nutxlffiun tax 


115. a b c d e 


116. abode 


117. a b c d e 


NN. Heternilhe whether an Individual quatlflef: 
for tiicbwe averaging and compute tax iifling 
achedule C 


118; ;i b e d e 


a b c d e 


120. , 'j r d e 


op; Rjiply the rules for addltlbna to tax by 
^ jj5_civli penalties whjchjay be 
p tl\lv> and the -imount of interr*»t duo. 


121. n b c d e , 

I 


122. <1 h c d e 


123. a b e d 6 0^ 



1 ' U OU 
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GENERAL QUESTIONS 



Questions 12A through i29 REPRESENT TASKS PERFORMED FOR ASSIGNED GASES IN 
ON-THE-JOB TRAINING. RATE EAHP TASK ACCORDING TO YOUR OPINION OF YOUR 
GENERAL LEVEL OF PERFORHA^NCE USING THE FOLLOWING SCALE: 

a . could hot perf or: task 

b. sut stahtial errors in perfbrtiiance 

c. few errors in performance 

d. fully sati.^f actory performance 

e. superior performance 



i2A. Complete and assemble all case file forms correctly. 



125. Conduct initial interview in a professional and courteous fanner, 

conveying and obtaining all necessary information and explanations. 



'26. Develop case with approved pre-examination plan , a logical sequence of 
audit steps, and significant issues identified and researched. 



127. Complete workpapers clearly and concisely, she -ng facts adequately 
developed and documented, ^^.^n. conclusions t;itl authority cited. 



128. Prepare Repx^-Lt Form properly with correct entries and ::urapu tat ions; 

and all necessary attachments. 



129. Conduct closing conference in a professional and 
explaining all findings nnd proposed adjustments 
his rights, and considering the taxpayer's point 



130. How did the classroom training as a whole prepare you for the 
job? • 



a . 

b. 

c . 
d. 



courteous manner, 
, informing taxpayer of 
of view. 



hot at 'all 
very little 
adequately, ^ 
very well CJ o 
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131. Are there tasks in the revenue agent job that ^hbul4 , have been trained 
in the classroom, but weren't? 



a. yes 

b . no 



Please use the rest of this page to explain what tasks, if any, you feel 
are not trained but should be. 



THANK YOU FOR COMPLETING THE QUESTlONi.AIRE. PLEASE. 
PUT ALL THE COMPUTER SCORE SHEETS AND THIS PAGE INTO 
THE ENVELOPE PROVIDED, AND MAIL IMMEDIATELY. . 
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Evaluation of Eevenue Agent Bntt S Claasroom Training 
— Directibna for OJT Coaches— 



The trainees you are etching in OJT are part of a Revenue Agent Onlt I 
eiasn aeiectad for a special evaluation study conducted by the nattonai 

office; The study is designed to measure the effectiveness of the 

training and to find ways to improve the training. It ^vtU^S ie^t be^sed 
to Jieasare the Individual trainee In any way . Several classes froa 
aeverai regions are involved in the study » and all results will be 
expressed as group averages and correlations. 



The specific purposes of this type of evaluation ere i) to deterwine if 
trainees can petfoti the job after can^leting the classroom trainings and 
2) to see if the right tasks Were trained for the Job; To acco^lish 
these objectiyesi ve mist obtain data^ regarding the trainee during 
classroom training and at the completion of DJ^; He have already 
received classroom data; the ftnclosftd questionnaire will provide us 
^ith all inf omation needed at the end of b«)T. 



Pleasef ill out the anclosed questtonnsire using the Optical Scanning 
Score Sheets for your responses; Fill out a separate questionnaire for 
iach trainee. There are 2 aections to the questiomuaire. The first 
section requires one score sheet; the second aection requires 2 score 
sheets. 

Each section has its own directions. Please read thcjB carefully. The 
first section asks questions regarding aitscssiaent of the trainee. The 
srcond section present'r Mb learned in claasroom trai-iing. and asVs 

you to rate the- i^rc^ru±u^ . the frequency they arc performed, hov weil 
you f«l the tn :^ was trained for then, and how well you feel the 
trainee initiail.^ ^rfori^d them in OJT. This section also presents 
tasks the trainee performed in OJT in association with his/her cases, and 
asks you to rate how well the trainee performed them. 

Please be candid with your responses, this questionnaire can only be 
affective in Improving the course if we receive honest answers. The 
trainee 'c? social security number is required on the ©core shrets so we 
can track classroom data to this OJT quest iomaire. After zh^ ^^"^l" 
matches the 2 sets of data, all individual numbers and reports will be 
inaccessible. Our purppsc, as stated «rlier, is to measure training 
effectiveness, not the individual trainee. 

It is equally important to the study that we get lOOX response f^oi all 
coaches. Please return all S score sheets and the iMt fsge ofjthe _ 
third section (requiritig a written response? 

IT^iU ^ . Do this for aach trainee you evaluate, ^e^trainae « 
be^g a&d t^ fill out a similar questionnaire for h f /herself bo we 
will have 2 aourcas of information. Thank you for « » 

completing this study. 



EKLC 
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SECTION 



:aUND DATA 



Fill in the biankfi in the fblloving afitmer: 



Hume : ' * 
Course: 
Tsftt Code: 



Evaluation of RA I Training 



Background Data 
Training Site 



Date 



left 
. corner) 



Part 1 

Copper 



Write the trainee identification number (aoclal security 
nuinber) iti the first section of the score sheet. Wt'its 
the number 'In the manner shown by the example on the 
score sheet. 



1. 



Are you completing th i s questionnaire as : 



trainee 
h. on-thc- job-instructor 



NOW GO TO QUICSTION 15 OJI 'iOUR SCORE SHEET AND ANSWER THE FOLLOWB^G: 



15, Rate the trainee based on his/her oti-t4va- ioi>-p et^4^riaaQC^ 

a. Unacceptable 

b. performance docs not meet in every respect 
the fully acceptable level described for c. 

c. fully acceptable 

d. achieves ftnd in some respects exceeds the 
fully acceptable level described for c. 

e. exceeds acceptable standards. 

16. Identify toe trainee's score in Revenue Agent Un5.^ 1 
ClassrooTD training (percent of objectives mastered) 



a. less thju. 70Z 

b. 70 - 8DZ 

c. 81 - 90Z 

d. greater than 90Z 



17. 




Please jive the training recomae ndatlon pr o vide d-J^ 



b. Degree of supervisiOD 

c. Degree of supervision 



a. Degree of supervision 



average 
minimum 




(Excarpts) 



81 



SECTION B; TASK RATINGS 

Fill in the blanks in the fbll. v: • r ^ 

Maae: fvaluatlbn of 'AA 

Course: Task Rg tings 

Teat Code: Training Site - 



Part 1 

(Page 1 upper 
left comer) 

Part^^ 

(Page 2ft : pper 
left cottier) 



Write trainee ^0 ir!entif ication huiober (social 
aecarity nu^er) part 1 on the first score sheet. 
Write the number in titic manner shown by the ezasiple 
on the score sheet. Do the same on the second score 
sheet in part 3. 



INSTRUCTIONS 

the tasks listed on the following pages arc the ones that 

were used to design Revenue Agent Unit I claesroom Training. 
Rate the tasks cccording to the trainee's performance in on- 
the- job training. Rate each task according to three 
different characteristics. 

FREQiJ^CY - How often did the trainee perform the task? 
ADEQUACY OF TRAINING * How well did the classrbon 

training alone prepare the trainee for the task? 
JOB PERFORMANCE - How well did the trainee initially 

perfdrtn th^ task? 

Record your ratings on the score sheet vith only 4^ne 

response for each characteristic under a task. As you 

record your ratings, check to awiV-e sure the numbera 

under each task correspond to the nuMbers on the score 

vheet. The questionnaire and the scdresheet -correapond to eact 

other by number as shown below with nuaber 1. * 



QUESTIONNAIRE 



SCORESEEET 



Task 


Frequency 


Adequacy 


— 1 

- - 

P^r f o man c e 


A. 


(D— — 




!3. 


B. 




5. 

1 


6. 


- 

C. 


7. 


i 

: 8. 

i 


5. 



' 10;, 



11. 



12. 



3. 

4. 

5. 

6. 

7. 

8. 

9; 
iO; 
11. 
12. 



ERIC 



UCOU) iU. ADSHERS ON 
CONPUtEli SCOltESiiEET 


I. irtit PvfltirH 

1. hrftiw^ r*i 5 if JjMW. 

1. NrUni^ r»rM( •( turn 
1. hft%mi ttr III CMN 


! n^^msf 

1. Taia N)ilrM Mliiff Npauli 

li Italil^ 
f. triiiiii im ihn i^Miii 

_ inrtiii iflMli 

f. tnliliq i«H«ii Im iMl 
d. tliUli) Mfi ilM laifMdt 

1. Cimltrt«ici fi ilaiiiiti 
irsUlig Itr ml 


mmmm 

?! ^l!«.H!ia!i.lllb__ 

^* Mitiiiiil ffftn ill i^timm 
i. If» tfiiti la H^ftniin 
d. IMifMlifiliitiiMitf 
a. Ifif lir pirtMiiici 

lii iii iM^f if tilt iu« 
imr kfi prftfiH} 


TASK 








A. LMit< ind elt( Zait ini Itiulitl itctionii 


1 I b e d a 


9 1 b c d 1 


V :• * c a 


1, Locittf ind dtf Rfvenuf tullngi ind court cim^ 


4, a b c d e 


5, i b e d ( 


5^ 1 b e d I 


I', tvurtliw thi tiiipiy^r*! corrtcUn \mi\lj 
uiin^ tl»f til tiblei or in rite "bchdules, 
v)ik^'Vfr If appUcoblc. 


7. a b c d e 


8. » b c d 1. 


Q i b c d a 


D. Coi^tt tbi prtpiirtint credtti for Eirncd 
Ihcdn ind/or Excfii HCA. 


10. a b c d a 


11. « b c d . 


19. t i i i • 


E. Detcriinf tht correct deductlofi for personil 
ixcipUdhi. 


13, a b c do 


14, 1 b c d 1 


15^ a b e d . 


r. Oiunplne the correct filing itttui for 
Ifidlvidttil iricoie m mm. 


. 16. a b c d e 


17. 1 b c d f 



18. i S « a . 


C. Identlff IteM of iron IncoM, ind recognlie 
certilh itaiiiCbry eicluiioiii froi iron incofie; 
ittchn |i(tii inherltincfi priiei, icholirihlpf 
Af f^Mnuttitni Hft InituraMe nrocfedfi ind 
thteffic oft Stite and wnlclpal obligations. 


19. a b c d e 


21?. 1 b f d ; 


21. . k c d » 


II Octcrtine lAfther in.expfnie li deductible 
in tlie coquet d( a tfido or buitneiit, and 
dilitlrigulih betveeii capltil expenditures 
dnd ofKatlng exfienNCfi. 


I,'!, a b c d « 


23. • b e d « 


24. • ^ f ' 

— «« 


j Detefiilne^ tk* drductlbtlltjr'of travpl ind 
traniportotlofi vxpenni!. 


35, A h t i t 


26. • h f J * 


57 a b c d 1 


1 Oetentw tKe dfauct|bili()r «f em«?rta|i«cni 
enieria|n»fnt lacllujr vKprmll turen 


28. n b c tl^ c 


29 i 41 b c d i' 


30, a b c d e 



ERJC 1^ 
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GENERAL QUESTIONS 



Qaestions 124 through 129 REPRESENT TXSRS PERFORHED m.^SI^^JED 
ON-THE-JOB TRAINING^ RATE EACH.TASK ACCORDING TO THE; GH^ERAL PEiU^OjkHAi^.C L 
LEVEL OF THE TRAINEE USING THE FOLLOWING SCALE: J • 



i. could not perfoztD task 

^fc. oubstanti«l errors in perf orrszincc^ 

c. few errors in perfonnance^ ■ . \^ 

d. fully stttiBfactory performance ^ 

e. auperior performance , ' 



124. Cdaplete and aasemHIe all case file forms correctly. 



125. Coiidnct initial interview in a professional anu courteous manner* 
conveying and obtaining all , necessary information and explanatiotis. ' 

126. Develop case with approved pre-examination plan» a logical sequence of 
audit Bteps.^and significant issues identified and reseaxched. ^ 



. ___ . " 

127. Complete workpapers clearly and concisely, showing facts adequately 
developed and docm&sntcd, and conclusions with authority cited. 



128. Prepare Eeport .Fornf-'4549 properly-with correc.. -rntries and computations 
and ail necessary attachments. - ' 

' ■» -. . * ■ 

t 

129. Conduct closing conference in a professional and courteous manner 
exi^laihing all findings e^ud proposed adJustmentB,^^tofor2i4n8 taxpayer of 
his rights, and co isljiering; the taxpayer^ a point of view. 



130. Hb: did the classroom ^training as a whole ^ r^spiprc the trainee for the 
Job? ^ . 

, ^ ^ ♦ a. not at all - , ^ 

,1 . ^ . b. very little ^ y ■ ; (. 

c, adequately f , 5 

di. very well # y. * _ . ' 
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131. Are there taaks in the revenue agent job that sbould* have been trained 
in the classrootQ* but veren*t? 



ft* yes 

h • no 



Please use the rest of this page to explain what tasks, if any, you feel 
Are not trained but should be. 



i 



i 



i 

i 

i 



■ 




THANK y( 
PUT MA^ 
TFE ENVl 


— . " — — • — " — ■ — ■ ' , — ' — 

)U FOR ee*£PLETING TBt QUESTIONNAIRE. PLEASE 
THE edSPUTER SCORE SHEETS /iND THIS PAGE INTO 
^OPE PROVIDED, AND MAIL IMMSDIATELY, 


1 
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EVALTJATio.. :t...TflODS AT THE EMPtOYHENT STANDARDS ADMtNIStRATf Gf^ 
OF THE DEPARTMENT OF LABOR 



The Branch of Training and Development at the Eraployment Standards ' 
Administration (ESA) is concerned with assessinj^ both the participant s • ^use of 
newly acquired skills (individual performance)^ and the irnpaqt of using new 

_ _ _ . _ _ _ ... _ _ _ _ _ ■ ^ . . ' 

skills on orpianizatibnal per^ ■^rni^iricer The two descriptions which follow 

' ) ^ - , - - ' - : 

represent each of these foe. and are typical of evaluauibris carried out at 

-4 • — ^ - - 

H55A« The first method preisented addresses evaluation of technical personnel 

such as investigators and claims examifi^fes^ and the second looks at office 

Tnana)a:ers. The methodological aT>prbnches iar^ adapted within ESA for each 

evaluation performed, to produce the specl/ic inf ortTiatidn need ad, ^ / 

The evaluation described in Part I was designed by KSA st- member Margaret 

Hetnslev, The evaluation described in Part II was implemented by Gloria 

Pe^irls tein, ^ ^ 



8S 

EMPLOYMENT STANDARDS ADMINISTRATION 
PART i 

EVALUATION OF DESK AUDIT TRAINING 



Evaluation of the tralnlnj^ of ajs^ents such as Invest ijs^ators and claims 
exaralners Involved needs analysis of trainees* skills, a task analysis, 
development of course obiectives based on the skill levels and iob task 
• reC|Uirements , an in--course evaluitton, and a supervisory ort-lob performance 
evaluation four to six months after the complet ion of the course. For the in- 
course evaluation, comparisons of skill-level achievement were made amon^ 
RToaps of trainees with different levels of experience prior to traininj2. For 
the follow-up evaluation, comparisons were made between ^roup needs before the 
course and supervisory ratings of post-course performance. The final report 
provided a summary of the results, a data display, and a synopsis of the 
raethodoio^. 
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GBSLERAL D£9CRiPtlGN OF THE EVALUATION METHOD 

Courses for invest l^^ators and claims examiners are evaluated in terms of their 
impact on the job performance of individuals. The EraTjloy-ient Standards 
Administration (E^) offers several courses for these persoiffiel; the one 
referenced in this review is the Desk Audit Skills Course (DASC). Other 
courses are evaluated in a similar manner with some variations. 
Equal Opportunity Specialists (EOS) conduct desk audits of affirmative action 
plans of those private industries which arie contractors to the Federal 
l^overnment. (Two examples of contractors are utility companies ard paper 
companies.) When the contract iexcieeds a certain dollar amount or the number 
of employees used in the contract exceeds a given number, then the contractor 
must abide by the requirements of Executive Order No. 11246. In doing so, a 
contractor submits an Equal Employment Opportunity (EEO) Report, which may be 
selected for review for various reasons^ When selected, the contractor is 
requested to submit an Affirmative Action Piani Oaring a desk audit, an EOS 
reviews thiis plan to identify areas where the contractor may not /be in 
compliance with the Executive Ofders The desk audit may be followed by an on- 
site review, in which the problems indicated in the desk audit are 
investigated. 

Clearly EOSs who conduct'desk audits must be well-heeled in knowledge of 
Government regulations pertaining to Affirmative Action. To qualify for the 
DASC they must already have had an introductory course in contract compliance 
and worked on the iob in a trainee position for three to twelve months in the 
supply and service areai Those in this area work directly or indirectly with 
the conditions private industry must meet when they ^supply goods or services 
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to the Government. 

The Desk Audit Skills Course was developed in response to a newly consolidated 
workforce drawn frotn eleven different Federal agencies^ The new group needed 
to be able to follow a uniform set of procedures in conducting their reviews 
and invest igationsi A needs analysis produced a number of proposals for 
training, of which the Desk Audit Skills Course was the most basic. The * 
course itself is nine days long and extends over a period of two weeks. Over 
a ten-month period, about 40 offerings of the course were provided, with 15-20 
participants per class. 

The evaluation of this course involved the following process. 

Identification of :iob competencies - The skills necessary to conduct a desk 
audit were listed in detail. This analysis was conducted through interviewing 
a small sample of incumbents, using a structured format. 

Needs analysis ~ A sample of prospective candidates for the DASC were asked 

detailed questions about their skills using highly structured interviews. The 

purpose was to determine if the individuals already t50SGessed the skills 

>■ 

required to perform the iob. The ' interviews were conducted by phone or by 

questionnaire. Some questions required a yes/no response, and others required 

explanat ions. Three examples are presented below: 

- Can you identify technical deficiencies in a Work Force 
Analysis? (yes/tio) 

- Can you' ident if y when a contractor's goals are realistic? (yes/no) \ . 
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- What the difference between systemic dlscrlTnlnatlon and 
affected classes? (detailed response) 

Employees were assured confidentiality and asked to be as candid as possible, 
in describing their skills. If enough people alf^^^d^khew how to do specific 
things, those tasks were eliminated as areas for eventual translation into 
coursi^ objectives. (See Attachment 1 for survey introduction and additional 
sampl^ciuest ions and recording format.) 

Task analysis - Expert practit loners were selected for an intensive two-week 
session of In-^epth task analysis. These individuals were considej-ed by their 
peers and supervisors as high perfdrmers in their Jobs.^ The tasks analyzed 
were those derived from steps one and two - skills needed for competent 
performance and which were lacking in the population under consideration. 
Individuals were asked for detailed explanations of how the ^oh is carried 
out, in terms, for instance, of what is particularly important among all the 
tasks, and which tasks are more difficult tVian others. 

Statement -of-^b^ ecti vgs - Course objectives were formulated using the 

ilk* 

information provided bv the task analysis. Then a second survey was conducted 
with a sample of Equal Employment Specialists -representing the eleven 
different ac^encies, to identify who possessed which skills and how frequently 
they practiced the skiilsi Thus the course was designed to provide skills 
which were needed to perform the work, and which wero not possessed by the 
prospective participants. Based on results from the representative sample, 
conclusions were drawn and generalized to tb^ entire population relating 
nature and extent of experience, and neod for course. 



In-course evaluation - Two types of evaluation techhiqiiesi were used within 
each Desk Audit Course. One was a "pre-test" and "post-test which were 
actually questionnaires in which EOSs were asked to indicatie how confident 
they were about their ability to perform specific tasks required to cotaplete a 
desk audit (see Attachment 2). The second type of evaluation technique was 

the use of an "Assessment Case" - a fictional Affirmative Action Plan 

^ 

simulating a typical situation the KOS encounters on re^;ular desk audits. The 

exercise took most of a day to complete and was the most direct measure of the 

performance level achieved by each EOS at the end of : the course. 

While the course was desired for EOSs with three to: twelve months of 
experience in conduct injs; desk audits, most classes had some participants with 
ho experience and others with several years of experience. Thus the evaluator 
analyzed the results of the two types of in-course exercises not only for the 
classes as a whole., but also by category of experience-. The primary question 
to be answered was. Did the three - twelve month group achieve the level of 
performance sought? Two corollary questions were^ Did the course enable those 
with little or no experience to conduct a desk audit satisfactorily? and Was 
the course of any benefit to those with more than n vear of experience? By 
analyzing the data for each sub-group^ the evaluator was able to draw 
conclusions about course effectiveness and participant population. The 
results confirmed that more experienced EOSs should not take the course, and 
that newly-hired EOSs should gain some experience* before being trained. 

Fol!ow-np-evaluation - Although the original course structure called for two 
trainers per , session, resource Constraints resulted in some of the courses 
being taught by only one instructor. Because the follow-up evaluation was 
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originally designed, however, to assess two-trainer sessions ^ the sample of 
participants selected for inclusion iti this evaluation was chosen only from 
those sessions- The sample was further narrowed by selecting two-trainer 

soBsions that were timely relative to the evaluation ef f brt j and selecting 

_ - ~ ' - - - 

students in those sessions vho met the criterion of three to twelve months 

experience (some classes contained more experienced individuals). 

The final sample consisted of twelve oarticipants from two classies. While a 

ten percent sample size would have been preferable (and appropriate, given the 

large number of employees taking the course), the evaluators chose to work 

with individuals for whom the course was originally intended, aud with courses 

of the original design, 

A survey questionnaire for supervisory evaluation of trainees was based on 
thirteen aspects of EOS work during a desk audlti These work areas were 
reflected in the ob.lectives and taught in the course* There was indication in 
the needs analysis that most trainees did not have these requisite skills, 
which support their inclusion in the supervisory evaluation questionnaire. 
There were five numerical ratings of each task. An example follows: 

Item: t?oS determines current employment patterns of protected group members. 

I 2 3 4 5 ^0 

Unacceptable Marginal Adequate Good Thoroufl^h ' Have Not 

- Observed 



Supervisors of the former trainees In the sattiple completed the questionnaires 
four to six months after the termination of the" courses this tine period was 
chosen because of the life cycle of reviews and investlJ^atlons (60 days) and 
the need to obtain enough samples of work for accurate judgments to be made* 
The evaluator let the supervisors know ahead of time that the survey would be 
mailed to themi With orpcanizat lonal support for such surveys, the 
questionnaire return rate was high. 

It should be noted that a pilot survey was conducted to test the 

quest lonnalre; . because the results of the survey showed no need to revise the 

questionnaire, the pilot group was Included In the final results. 

Responses were collected and the average supervisor response to each item was 
calculated. Evaluation conclusions were drawn by comparing group needs biefbre 
the course to group averagies from supervisory ratings after the course. Group 
comparisons could be used because the course wais bathed on the' results of a 
needs analysis of prospective participants, and because the sample ^used for 
the evaluation met the course design criterion of three to twelve months' 
experience , on the 1 ob before course attendance. 

For both the In-course evaluation and the follow-up, a final report was 
producied, describing the purpose, process, data analyses, 'and results and 
recommendations. The evaluation results were provided to management and were 
used to verify adequacy of course design and content and aopropriateness -of 
participant population.^ 
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ANALYST 5; OF EVALUATION METHOD 

Condt£lons_£or use 

~ Top management support for conducting evaluation efforts^ including <^ 
survevinp outside the course environment. 

- Training office requirement that, a portion of staff time be devoted to 
build inK in evaluation as part of the course deslpn (Including obtaltilnfic 
outside resources If needed;. 

- Ability of training function to work with client, to develop an 
understanding of the nature of the job and the level of performance 
expected, and to gain cooperation for the evaluation. "Ability" includes 
both skill and credibility. 

- Front -end analyses^ to develop Information on skills needed to do the job, ' 
skills actually possessed by potential participants, and detailed 
descriptions of skills for which training is appropriate; needed as basis 
for course design and evaluation design. 

Resources requlf^ ^. 

The main resource needed Is the trainer* s expertise In developlnj^ needs 
analysis and task analysis processes, and designing the course based on the 
resulting Information. Also needed Is some^ ability In designing: the 
evaluation process, including c uestionnaire development and sample 
selection. Experts outside the tralntpg offlfje can be used^ as advisors to the 
evaluator. The follow-^R questionnaire flows from the pre-course efforts. 
Other resources Involve the time of those supporting the course planning 
(e.g., the practitioners participating In the task analysis), and those 
filling out the follow-up questionnaire (the supervisors). 

i . 1 - 
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The seiectlbn of a sattiple population instead of all trainees taklnR the course 

cuts down conslder^hly on the data collection and analysis.' 

r' .: 

W atuge of information produced 

Averages of raw data were taken from ratings of the In-course aSd follow-up 
.questionnaires. Average scores from the simulation exercise at the end of the 
• course were derived. The evaluator determined if perforiance was occurring; to 
a level satisfactory to the supervisors of . the participants. Because the 
course objectives are based on a task analysis of the .lob. if results from any 
of the assessment devices point to less than acceptable performance, the 
evaluator can examine the relevant portions of the course to determine what 
changes should be made. 

Advantages 



-Course content based on a needs analysts which indicates the trainees' 
strengths and weaknesses, and a task analysis which ties course oblectives 
to job tasks. 

- Confidentiality assured, which encourai^es honest responses. 

- The use of a sample population which consideranly decreases rhe amount of 
^resources required. 

- Ability to pinpoint the specific portions of the ccurse which may need 
changing, based on evaluation findings. 

\ 

Limitations ' . . 



-In this case, resource and administrative constraints, which oroduced a , 
small sample size for the follow^p, and which limited generalization of 
conclusions. 
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- The timie needied for the course, design and development (six months in this ' 

_ _ _ _ _ .... _ 

..^ case), and tl?e time period needed between the end of the course and the 
follow~t>p (four to six months)."" Without administrative delays .and ^Ith^a 
simpler situation, a shorter titne peribd could- be-v^xp§i:t;ed. 

- In this case, the link was not made between end-of-c6urse learning and post 
course peffornance impr ovenient on an individual basis; this comparison 
could be made vri.thin the design as described above, 

Reference 

Brendan Branley, Chief • 
Branch of Training and Development 
Employment Standards Administration 
Room S3013C 

Department of T^abor • " „ 

200 Constitution Avenue * NW 

Washington, DC 20210 

(Phone: (202) 523-7036) ^ 
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SURVEY INTRODUCTION 
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- ^ SURVEY INTRODUCTION . . 

: y : . . 

I am from the ESA trainlm office. I'm working with dFCCF\to 

develop a desk audit skills course that will^be primarily for EOS' who had 
less than a year of field experience in conducting supply and service reviews 
before consolidation. 

Right how, we're interviewing EOS' who are likely' to be in the course to 
find out exactly what area^ the traininff should cover. By doing these 
interviews, we'll be able to pinpoint the skills and knowledge that KOS' 
themselves say they need\^and we can custom-tailor the course to meet your 

needs. The survey gives you a unique chance to tell us exactly what areas you 

.1 _ _ _ ._ . . . . . " Y - " " ^ 

•need to know more about in order to conduct 'a quality desk iaudit. 

lid ybu have any questions so far? 

.•V ■ ' ' % j 

The survey will work this way: 

I'll read a series of questions about specific parts of a desk audit. 

: Some will ask whether: you can do a specif ic. task and some will be open-ended 
questions' that require m^re than a "yes" or "no"'answer. The survey will take 
ab out an .hoari ^ ^ ■■ 

Please be as candid as you can in answering the questions. If you can't 
do something, please don't hesitate to say so, jsince the only way we'll know 
that you need certain items in ^is course is if you say so. By the same 
token, if you can do something, . please say that* If enough people .already 
know how to do specific things^ V7e V7bn't waste course time on that item.- And 
since we're interviewing onlya^ small number of all the 'EOS' who wtii be 
taking the course, your answers are very important. 

' I want to assure you that all survey results will be kept confidential. 

. W^'ii use the results only to develop the course. 



Are yoa ready? 
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SURVEY INTRODUCTION - PAGE 2 

First, let tne ^et some background data: AREA OFFICE 

(1) Where did you work before the consolidation? 



(2) Did yew have any experience In conductln;^ supply and service reviews 

before the consolidation? 
YES 
NO 

(3) if so, how much experience did you have? 



(4) If not, what type of work did you do before the consolidation? 
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YES 


NO 




bo you know what criteria a _ 
contractor must meet in order 
to be required to have a 
written ftftP? 








bo you know-how to verify 
that a con tr ac tor is svb^ ect 
to 11246 requirements? 






i 
1 

1 


if a company will not tell 
you whether or net it meets 

the criteria for 11246 

coverage, do you know what 
action to take? 








(a) bo you generally look at 
case fileiof previous 
reviews? 








(b) If so, what significant 
data would you examine 
in the previous file? 









If you don't receive all the 
data requested in an AAP 
package, do you know what to 
do? 



No question about reasonable effort 
since^ 4>?e— already know it's a need ' 

/ Can you determine whether a 
contractor's Work Force Analysis 
meets the criteria for 
"responsiveness"? 



• i 

I 
! 
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ATTACHMENT 2 



SKILLS ATJDiT OUKSTiCNNAIRE 
(IN-COtJRSE) 



ii-d 
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(Excerpt) 



Today's date 



Experience in 
S&S Reviews 



Ques ti, onna i re 



Please use the scale below to indicate how confident you ^are 
right now that you can do each of the items listed below 



i.e 

Not 

Confident 



2\B 

Confident 



3!0 

Moderate ly 
Confident 



To 



Confident 



"s! 0 

Completely 
Confident 



Rating 

1. 



3. 



4. 



5. 



7. 



8. 



Determine if a contractor's AAP submission meets the 
criteria of "reasonable effort" to comply with the 
regulations for submitting a written Affirmative 
Action Plan. 

Determine if a Workforce Analysis is acceptable and 
if it is not, identify what deficiencies are present, 

Determine if a contractor's listing of ^jbb groups" 
are acceptable. 

Determine if a contractor's availability estimate is 
acceptable. 

Determine if a contractor's goals and timetables are 
acceptable^ 

Determine if a contractor has made acceptable 
progress toward meeting the past year 'is goals. 

Determine if a contractor has made acceptable 
progress toward meeting the current year's goals. 

Determine if the contractor has made ah acceptable 
statement of "good faith effort" in instances where 
goals have not been ^t. 
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EMPLOYMENT STANDARDS ADMINISTRATION 
PART li 

EVALUATION OF FEDERAL EMPLOYEES' COMPENSATION 
DISTRICT OFFICE MANAGERS WORKSHOP 

SUMMARY 

The evaluation addressed a course for district office managers on how to use a 
new Management Information System to analyze operational problems and to track 
the effectiveness of improvement efforts. During the course, the managers 
chose their own real problems to tackle when they returned uo their jobs, and 
developed a plan of action, the evaluation involved the training office's 
tracking monthly reports and interviewing managers at quarterly intervals, to 
-ascertain the extent of action plan Implementationi 



o 
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GENERAL DESCRIPTION OF THE EVALUATION METHOD 

Federal Einjployees' Compehsatton Dlstrlci: Office (FECDO) Managers were trainied 
to use the Mana^^ement Information System (MIS) as a management tool* The MIS 
consisted of monthly summary data reports reflecting the amodht and rate of . 
Diroduction, unit outputs, and other workflow measureis for each district 
Dffice. Assistant Deputy Coinmis si oners. Chiefs of Claims, Supervisory Claims 
Sxaminers, and /or System Managers from fourteen district offices attended the 
workshop, which was three and a half days long. The purposes of the- workshop 
^ere threefold: . \^ 

!•* Managers learn how to examine MIS reports as indicators of operational 

problem areas* \ 
2* They identify their own problem area^i in district office performances 
3, They develop their own goals and strategies for resolving specific 
production problems. These goals were to be the benchmarks against 
which future change in performance would be measure^i. 



'he reason for developing the workshops in the first place stemmed from the 
urganizat ion's introduction of a new MIS, and the resulting resistance by some 
lanagers in accepting and using it. They saw the system as producing 
ir oductivit^' measures which were "mechanical" in nature and which could be 
:sed primarily in a negative way* The organization's obiective for the new 
:ystera was to maintain an accurate data base, by improving the q-tality of the 
ata and by Incf easing managers' understanding of how MIS results will be used 
o measure managerial performance. 

1 5 
J. ^ v> 
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As a irieans of oviercoming the reactions of managers to the new MiS; the 
emphasis of the course was on showing managers the payoff to tAem' ^y keeping 

__ 

good data to show them how the system coaid be a valuable tool as they 
managed their own unit, and how it could be used to demonstrate their own 
successes to top management, V , 

■ * . . 

The course was designed to be a one-time of ferlng . (given enough times to cover 
all the district managers involved)* The purpose of the evaluation was to 
ensure that managers learned how to adapt the MIS to their cwm needs and that 
they followed through in applying what they learned to measure and improve 
their own operation. 

The following steps comprised the evaluation process. The MiSo reports, which 
were already in place, were also used for evaluation purposes. 

Explanation of process and implementation of training - Workshop coordinators 
explained that the trainees would develop performance goals for their units 
and that the Branch of Traininj^ and Development (BTD) would monitor 
achievement of these goals as a way of paging worlcshop effectiveness. The 
National Office did not stipulate which goals to develop, nor would it monitor 
their achievement. 



The training staff provided instruction in using the MIS as a diagnostic 
and monitoring tool. Working in teatnis, the managers chose to examine a 
situation in their units where problems were occurring but where the causes 
were unclear. Ah example was: Being able not only to determine why 
compensation cases were not helng processed within a certain time, but also to 
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discern which classes of cases were hetni processed at slower rates than 
others, and thus beinj? able to address the specific problem. The manaj^ers 
learned to adapt the organization's* Management Information System such that it 
wouid produce the data needed to analyze SuCh situations. They could then use 
the MIS to continually monitor their organization's performance and determine 
the effects of efforts made to improve these problem areas. 

Each district office picked one to three high priority problems which they 
wanted to address, arid set productivity improvement goals. Others in the 
class critiqued the plans. The training function retained copies of these 
plans for the follow-up process. 

Collection and display of data - The training office followed up with the 
managers to check on the progress of implementing their plans through use of 
the reports generated by the MIS. The evaluator obtained product: vity figures 
from the MIS on a monthly basis, then interviewed managers quarterly to 
discuss v>rogress and problems. The follow-up continued over a nine-month 
period. 
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the following are examples of specific j;bals developed by the managers during 
the workshop and the changes in performance levels over time. 



District Office 
Goal / . 



PHILADELPHIA 



Perfptinarice Level 
at Time of Workshop 



Reduce inventory of caaes 
under development to 8,000 

SEATTLE 

Improve percentage of 
trautratic cases adjudicated 
in 45 days 

DENVER 

In<*rease percentage of 
non-traumatic cases 
adjudicated in 150 days 
to 75% 



9,400 



46.3% 



Performance Level 
After Workshop 
(minimum of 6 months 
afterwards) 



4,713 



88.3% 



52.5% 



78.7% 



As can be seen in these examples, managers made progress as intended tn 
achieving productivity improvement Si The evaluation was in the form of 
assessing the extent to which managers carried out their plans of action. 



Report of results ~ The final report of course effectiveness was in the form 
of a memorandum to management, with attachments displaying the type of data 
Illustrated above. . The memo explained the purpose of the workshops and the 
process;, of evaluation. Conclusions V7ere drawn about geheralizable features of 
the training, such as the effectiveness of using goal-setting as an integral 
part of the training program In order to link skills training with program 
results i 
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ANALYSIS OF EVALUATION METHOD 

Conditions for jse 

- A Management Information System with computer-generated reports, and with 
the capability of being adapted to produce non-routine datai 

- Organizational support for collaboration between line managers and 

the training function to implement the new MIS, without^ top management *8r 
need for giving direction and oversight while the learning process occurs 
and follow-up data is collected* 

- Influence and control by the trainer over a learning process which 
spans pre-course planning (to ensure that teams of employees attend 
the training)/ in-course goal-setting (related to actual job needs)^ 
and post-course follow-up (on a fairly regular and long-term basis). 

- The participation in training of a management group with the decision- 
making authority, to establish and implement goals. 

Resources re<qu4red 

During the training itself, managers began applying their new knowledge of how 

"The MIS couW~be~used^ select-lng--acltual- sltuat ions , t p 

address, by setting performance goals, and by planning the implementation of 

their goals. Because the training was so directly linked to job performance, 

_ _ _ . . o _ . 

with the groundwork laid in the course for transferring learning to the job, 

the evaluation process itself required few resources. The training functi(^n 
reviewed regularly-generated production reports for changes in areas in which 
managers wanted to make improvements. In addition, the quarterly interviews 
with district office heads, ' carried out by telephone, required no more fhan 
half an hour each. 

It 
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Natute-of inf orraatt^^rxxixicecL 

,;. J — • 



The basic evaluation information produced through the follow-up was the extent 
to which managers accomplished the plans they deve3;oped during, the training 

y~ 

program; The evaluators could thereby assess achievement of the training's 
objective of enabling managers to use the existing data base to improve their 
ope rat ions i ' . ^ - c - 



. AdvaJrt^ages 



- The evaluation was based on plans which the district office managers 
-formulated during training and which were* directly related to their own 
situation. 

- The objectives in the plans were in concrete terras and thus accomplishments 

. ... _ .. ._ ■ • 'i ■ i 

were easy to track. 

- The evaluation could be based on the MIS reports and thus did not 
require additional record-keeping, 

- The fact that district office performance was not monitored after training 
by top management, but rather by the 'training office, made the process less 
threatening to managers. 

Limitations 



- The plans contained various types of goals ^ and thus comparisons of course 
effectiveness had to be made in terms of individual goal achievement not 
group results. Any summing across the Class would be limited to numbers 
of participants achieving or exceeding, these heterogeneous goals. 
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Reference " 

Brertdan Branley, Chief 

. 

Branch of Training and DevelopTOsnt 

Employment Standards Administration^ 

Room S3013C 

■# 

Department of Labor 
200 Constitution Avenue, NW 
Washington, DC 20210 
(Phone: (202) 523-7036) 
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DEPARTMENT OF THE NAVY 
NAVAL RESEARCH LABORATORY 



EVALUATION OF PORTUGUESE LANGliAGE COURSE 
(FOR TECHNICAL 'AND SCIO^TIFIC STAFF) 

SUMMARY 



Scientists and technicians at the Naval Research Laboratory took a basic 

i 

Portuguese language ccxirse before embarking on an assignment with their 

Brazilian Navy counterparts. The follow-up evaluation involval bringing- 

■ . - ' - - ■ / - 

together the group after they completed their next assignment to discusS; the 

" ■ ■ ' ■- ' ! • 

value of the course to their work. The semi-structured group interview 

process yielded specific examples of individual ^and group changes on the job 

after the training and suggestions for further training, \ 



"2 J 



Ill , 

" ■ /' 

GENERAL BESCRIPTICTI OF THE HVALlfATiON METHOD v 

Scientists, technicians, and support personnel in a Systems Research and , 
Technology groap at the Naval Research Laboratory (NRL) are working on long- 
term research projects wirh the Brazilian Naval Research Lab; During t>fo 
previous deployments of the Americans to Brazil, both aboard a research vessel 
and on land-, it became apparent that a language harrier was interfering with 

effective and safe project work; the limitations of hand signals anS. brpken 

_•_ _. _ _ • ■__ '^.'v - 

German and English were evident. One of the; project teade*ts informed the: 

training officer that he and his men were experiencing these problems^ The 

decision was thus made to design a Portuguese language course to meet the 

general as well as specilPic* (technical language) need^ of NRL's project staff. 

The course development and evaluation process involved these steps: Pre- 
course survey and course development; course implement at ion and post^course 
survey; pre-depldyment briefing; and poit-ndeployroent debriefing. The emphasis 
in the description below is bn- the p ost -deployment debrief ing. 

Pre -c our^e- survey ^nd-coorse d evel opment - A pre-course evaluation determined 
th^t' none of the participants had any previous formal Portuguese training; in 
fact only about 20% had any foreign language background. The individual 
chof?en.to provide the course was a linguistics professor specializing in 
Portuguese at a local university. As a native Brazilian^ the instructor w^s 
cognizant of the linguistic and cultural problems that arise between Americans 
and Brazilians; 
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The iniStrtictdr and^NRL's training specialist worked closely together to ensure 
that the course met the special technical and scientific nee<is of the project 
group. A coarse was designed to focus on vocabulary building, conversational 
Portuguese, and technical and scientific words and phrases specific to the 
group's specialty (oceanography) i 

Course impleTnent at ion aiid post-cotirse survey - The language training consisted 
of twenty sessions given over a ten week period. The participants met twice a 
week for two hours during duty hours. The classroom training was augmented 
vrlth home study in the form of audio cassettes. A post-training survey was 
used to obtain participant feedback on the course achievements^ difficulties, 
and restrictions, and to solici.t suggested changes for possible future 
Portuguese training. 

Pre-deplbyment briefing ~ Following the actual language training the group 
members were briefed by State Department officials on the current political 
and socioeconomic evients in Brazil. The session was recommended by NRL's" 
trainer to supplement the language portion of the training. 

Pbst^eployment session - As the main obiective of the course was to improve 
the jB^oup's performance while on deployment in Brazil, the final course 
evaluation could not be held until the group returned from the Brazilian 
deployment^ five and one-half months after the completion of the course. 'The 
individuals spent varying amounts of time (from two to six weeks) in Brazil. . 
As before^ time was spent aboard a research vessel as well as on land. 
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■ The post-evaiaatlon was held at NTRL during duty hours, the debriefing took 
about one hour and was held by the NRL training specialist who co-developed 
the course. The instructor was not present at this interview. However, when 
carrying out this type of evaluation the instructor's presence is optional; 
the evaluator needs to decide if the instructor's ptesence would help or 
hinder the information-gathering process^ Fifty percent of the class 
completing the course were present, as well as two individuals who due to .1db 
commitments were unable to attend the Portuguese language training. These two 
individuals attended on their own because they felt they had some significant 
contributions to give at the debriefing. In fact they acted as an informal 
control group. 

The session was structured only to the extent that the^ evaluator had developed 
a list of general questions to promote the H^oup discussion. The questions 
included these: 

Was this trip any different than previous trips? 

- Was communication with your Brazilian counterparts easier or 
more difficult than on previous trips? Why is that' so? 

- Has the language training been of any value? If so, how? 

- Would you take a second, more advanced course in Portuguese? 

- What changes would you ^ like to see in the format of the 
Portuguese language training? 

-T Would the individuals who didn't attend training attend a 

_ . _ _ - . 

training course In the future? 

^ How can training be of further assistance to your group in 
this specific area? 

JL U 
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The discussion was allbwisci to flow frora these questions. In some cases ^ the 
group addressed the questions before being asked^ anticipating areas of 
concern such as course Impact and heeded changes. The brains tormlhg effect 
fostered by the open discussion enabled one cbmraeht or suggestion to lead to 
others and likely produced more Information than would have been produced 
through Individual Interviews- 



Analyal8-^£^x »lldw-up feedback s - Participants' comments were noted by the 
evaluator during the debriefing and summarized into the following four 
categories: 

1. Suggestions regarding original Portuguese language trainings 

2; Suggestions regarding future Portuguese trainings 

3. Comparison of experiences between project staffers who had 

taken the language training and those who ?iad hot, 
4i Behavioral change which occurred as a direct result of the 
training. Examples Include the following: Americans and 
Brazilians were able to work more closely during the last 
cruise, as both groups practiced their languajEje with each 
other (the Brazilians had taken English language training 
between cruises); thus language barriers were eased. 
Results included not only greater ease and efficiency in 
carrying out pioiect work, but also greater camaraderie 
ard rapport. One further example (perhaps the most impor- 
tant,) was the Increased group safety due to foreign 
language ability, reducing the potential risk of acci- 
dents at sea with explosives and heavy equipment. 

1 Q : 
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- The following changes were made as a result of the 



information gathered in the evaluation process: 

1. A second basic Portuguese language trairiinjD; course would be 
conducted in a six week format for project Tnembers not able 
to take the original training and for new project staff. 

2. A new iriterroediate Portuguese lariguaiRe training course 
would follow the basic course; the course would run bight 
weeks and would enhance the ability to converse* as well as 
serve as a refresher for staff before they departed for 
their next deployment. 

3. The two new courses wbulc) contain certain content changes 
based on the evaluation results. 
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ANALYSTS OF EVALUATION METHOD 



Cggdlt ions -f or uBe 

- Ability of evaluator to re-group course participants; this ability depends 
on the comraitraent of the participants and their managers to the course 
and its improvement, as well as the geop^raphical dispersion of the parti- 
cipantsi (If necCTsary^ the evaluator can assernble a portion of the 
class and individually interview the remainder;) 

- Buiidinj? in the idea of a foilow-up course evaluation at the time of the 
course offering, to Rain support of the participants for the follow-up* 

Resources required 

The primary resource required is the time of the evaluator to conduct the 
follow^p interview and analyze the qualitative data, and the time of the 
p-»rt icipant s to attend the follow-up inter^^iew. Another important resource is 
an evaluator with facilitator skills to ensure the development of a dialogue 
that will produce usable information* 

/ 

Nature of information produced 

Due to tHe loose structure of the group interview^ the information is 
qualitative and heterogeneous. Perceptions of the individuals about impact of 
the training on job performance can he obtained during the discussion^ as well 
as any consensus that the group may be able to reach* Comments can also be 
solicited on the course process and content. ^ 



Advantages 

- Group process stimulates individuals to remember specific examples of 
behavioral change* - 
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-Where additional information exists (e.i^*, pre- and post-course ques- 
tionnaires) and indicates that fbllow-nip training may be heeded, a 
- foiiow-up interview in the^ form of a group discussion will allow 
specific needs not covered by the traditional evaluation process to 
surface and be discussed. This can essentially turn into a second 
needs assessment process. 

- If the group includes individuals who have not participated in the 
training process, but who had to function side by side with those 
who were trained, the evaluator has a built-in comparison group; 
discussions of the relative value of the training can occur between 
the two groups. 

Limitations 

— s 

- Enough time must be set aside for conducting interviews as well as the 
interpretation of the data gathered in the process^ 

- May not be able to bring together a sufficient number of trainees to 
develop a meaningful group discussion and useful feedback. 

- The semi -structured nature of the group discussion may produce dialogue 
which is not directly focused on the issues at hand; oh the other hand ^ 

* if the process „is too structured, the atmosphere may become too restric- 
tive to allow the free flow of information, ideas, and feedback. The 
evaluator^- or whoever is conductlpg the group interview^ n^<ids to strike 
a balance between the two situations. 

Refer^Jfee^ 

William H. Brown Hi 

Naval Research Laboratory (Code 1842) 
4555 Overlook Avenue; SW 
Washington, DC 20375 
(Phone: (202) 767-2956) 
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U.S. OFFICE OF PERSONNEL MANAGEMENT 

THE PARTiCIPANT ACTION PLAN APPROACH ^ 
A PEN ERIC FOLLOW-UP EVALUATION METHOD 

SUMMARY 

The Participant Action Plan Approach (PAPA)^ a generic evaluation method 
developed by the U.S. Office of Personnel Manapfement , is used to determine the 
extent of behavioral change which has occurred as a result of participants' 
attending a training course. At the end of training, participants fill out an 
actioh plan - a -list of changes they want to implement when they return to 
their jobs. The evaluator follows up with the participants several months 
later, through interviews or ciuestidrinaires , to obtain specific examples of 
change, as well as other data such as factors interferring with attempted 
change and the impact on the organization of new efforts made by 
participants. In addition to an outline of the method,: the description below 
includes specific agency applications which demonstrate some of the variations 
of the PAPA technique. 
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^Q^ERAL DESCRIPTION OF THE EVALUftTION METHOD 

The Participant Action Plan Approach (PAPA) is a method for assessing the 
extent of - individual behavioral chan)?e on the job due to participation in a 
training course or other type of formal learning experiences It is a flexible 
technique which can he adapted to a variety of courses, organizational 
situations, information needs, and available resources. The method was 
originally developed particularly for course subjects vjhich are difficult to 
measure through such means as testing, observation, or collecting productivity 
data. It was also designed such that it could be implemented after a course 
had been developed . and put in place. 

PAPA is based on the concept of participants' developing action plans - 
statements of what they will do differently when they return to their jobs. 
The plans are to be based on the training program completed. The training 
evaluatdr then uses the plans to follow up with the participants several 
months after course completion to obtain information on what behavioral 
changes the participants have actually made. While the notion of action 
planning is not new or unique, PAPA takes that process one step further and 
uses it as a vehicle for evaluation of training. 

the basic approach consists of five steps: Planning; ih-course activities; 
follow-up activities; analysis and conclusions; and final report, these steps 
are described below and include some variations on the theme. An approaoh to 
collecting behavioral change information without the use of action plans ±c 
also noted. Finally, brief examples are provided of the use of PAPA in 
different agency settings. in 
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The Baste Participant Action Plan Approach 

Development of a plan for PAPA - In this first step, the evaluator determines 
the specific actions to take to Implement PAPA, given the organization's 
Information needs and the resources available. The decisions are made In 
concert with the "client," the personCs) for whom the evaluation information 
is being provided (which in some cases may be the individual doing the 
evaluation). The questldnis to be answered by PAPA are determined^ questions; 
such as: What happened on the .1db as a result of training? Are changes that 
occurred the ones Intended by those providing the course? What may have 
Interfered with participants* trying to use on the job what they learned in 
the training? The specific design features for uklng PAPA are selected at 
this time, such as whether to follow up with participants by interview or by 
questionnaire, 

£l ondu c t -ri£^ n-course ^r±vi£ies - This step consists of two stages. At the 
beginning of the training, participants are introduced to the idea of an 
action plan and are asked to consider throughout the course what they might 
want to dn differently on their jobs as a result of the training. Then at the 
end of ihr training, participants are asked to write an actloT plan - a list 
of new activities, related to their learning experience, which they plan to 
try when they return to their johsi The participants are encouraged to . 
express their action items in specific behavioral terms. A portion of an 
action plan might have such statements as these: 
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i plan to: 

1. Talk more opeiay with my employees when 
things go wrong. 

2. Involve ray employees more iti making 
decisions about our shop operations. 

Forms can be provided at the beginning of the course which participants can 
use throughout the training to record ideas, for changes on the job and as 
guidelines for writing action plans (see Attachment 1). An action plan fonn 
is used at the end of training for participants to record action Items; 
written guidance can be provided to the group to aid th^m in developing items 

(see AttachTuent 2). Options for the in-course step of PAPA include: Setting 

1 - — 

aside time each day for participants to record ideas; and having pairs of 
participants discuss their plans with each other as a way of ensuring the 
plans are realistic and clear. 

The evaluator keeps one copy of the participants* plans, to use during the 
fol low-nip. 

Follow-up with-partictpantSN - At a planned time after the training (usually 
one to six months later), part Ipipant.^ are contacted, either through interviews 
or questionnaires. They are asked questions about implement ation of action 
items, other new behaviors attempted as a result of attending training, the 
effects that their new or changed behaviors have had on their work 
environiTK^nt, and what if any problems have occurred in transferring what they 
learned to the job. 
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. ^ ^ , \ J 

If /Interviews are u^ed, they can be done in person, by phone, or thrbaghia 

combination of the two.^ If the Interview approach is used, a go o3 policy to 



follow is to send a 



reminder letter to participants asking them to think aboat 
/ the implementation df th^ir action plans and indicating that they wiil be 



contacted shortly (see Attachment 3)i Even for telephone follow-ups, an 
appointment should bfe made with the. participants for the interviews. A seiSi- 
structured interview jformat can be used (see Attachment 4),lwith a copy of ; Ithe 
action plan in front |>f the interviewer (and preferably the Iparticipant as t 
weliyi The key to the use of PAPA as an evaluation tool is W obtain specific 
information about behavioral change. Probing, asking for specific examples,' 
and having the participants compare performance pre- and postptraining^ are , 
techniques to elicit information useful in assessing course effectiveness. 

. " : 1 t 

The two methods of interviewing each have their advantages. Ip-persbn \ 

1 : . ■ ^ 

interviews, for instance, enable the interviewer to react to non-verbal cues, 5^ 
, to eistablish rapport, and to see samples of work produces related to . j 

behavioral changes i Telephone interviews enable the interviewer to reach 1 

__. ; _ _ 1 ' _ 

geographically dispersed participant^ and may in some cases put | participants 

more at ease if they are describing iiss than> successful resultJ of attempted 

' : ^ \- ; ■ \ ' ' 

on-job changes. \ / 

■ ; \ _/,. ' 

the questionnaire approach to following up i^s iiseftil if time is riot available 



to conduct intervtewsi On the other-hand ;-th^ problems of using \ 

' - - - \ '■5' 

questionnaires apply here as in other situations - return rate can be low and 

• ^ : ■ . \ ^ . ; _ _ _ ■ ■■ 

responses are likely to be at different levels of\. detail and compljetenessi 

_. _ ■ \ \ \ 

Given the criticality to PAT>A of specific examples \of behavioral change and 

r ' ' • - -1 

the effectiveness of methods such.. as: probing to obtain such examples. 
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interviews tend to provide better quality data. Examples of l.is truments for 
the questionnaire approach are found in Attachment 5* 

Other variations in the follow-up step, besides the basic method of collecting 
data, include these: 

- Contact a sample of participants rather than. an entire class. 
Thik is particularly appropriate when interviews are . 

used and the participant population is iar^e. If questionnaires 
are used, the whole class should be contacted unless the group 
is of considerable size; 

- Following up with supervisors, subordinates, and/or peers of the 
participants. This effort can provide- corroboration of participant 

information, add examples of. change, provide a basis for possible 

_ -J 

future organizational consultation by the trainer if dircrepahcies 
appear among the various respondents, and increase overall the 
richness of the data. The organization's clima^ei however, may 
make such contacts appear threatening to the participants and 
may inhibit what participants develop for their action plans. 
Furthermore, the time for such additional cohtacts may not be 
available. Finally, evidence has shown that^ with proper 
interview techniques, participant self-repdrts of change 
match the reports of others working with the participants. 

- Reassembling the class to fill out the questionnaires. This 
approach can ^ensure a high response rate If: most members of the 
class are able to meet. It also provides an Opportunity for 

a group interview to discuss the use of new skills and knowledges 
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on the .1ob; Partlcuiarly if the partlclpatitis * jbbG and work 
Bettin;?8 are relatively horao;?eneoa8 » a discussion may trigger 
more examples of change and ideas for program iraproveinent 
than would be generated by each person responding separately, 

- lr»voiving the participant's supervisor before the trainings ideally 
supervisors discuss beforehand the training program an employee is to 
attend, to clarify expectations about what can be learned and used 

on the .i6b» Particularly if the learning obiectives are clearly 
performance-based - related directly to the work - pre-course 
supervisor/ employee discussioris will set the stage for learning 
and for the development of an action plan immediately relevant to 
the job and likely to be supported by management. 

- Tailoring both in-class action planning and follow-up questions to 
course objectives^ modules, subject are as ^ or some other structure 
which would enable participants to focus systematically on each 
course segment or goal^ and would enable the evaluator in turn to 
collect and analyze information along p re-determined and easily 
usable lines. , ' ^ ^ 

Arialysig of findings and development of cbnclusibns - The fbiirth step^ bf PAPA 
irivblves. sort ing, categbrizing, and displaying the data cbllected in order to 
shoti^ the iextent^arid type bf change". The infbrmatibn can be displayed in the-.^ 
fbrm of descriptions of behavibral change; it can he summarized numerically 
(e.g., how many bf the class participants changed in certain ways ) ; or it can 
be reported using a combination' of narrative and numbers. 

c ' 

The following are the kinds of data that can be analyzed: , 
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- The action Items themseivei. Do items match course intentions? 

: - New behaviors dlspiayid on the Job. How inuch were they related to 

interred changes? How closely were changed behaviors related to \ 
coarse objectives? How many people attempted what kinds of change? 

- Outcomes of behavioral change. What results occurred when partici- - 
pants attempted change? How did changed behavior impact the organi- 
zation? . .; . 

- Judgment of outcomes. .Was the result of the behavioral' change and 
: outcome positive? negative? mixed? If outcomes of change were not 

positive, the nature of the training program needs to be 
re-examined i 

- Problems and concerns. What harriers or difficulties ^id partici- 
pants encounter when attempting change? Does the course need to be 
modified? Should the training staff take other actions, such as follow- 
on support or consultation with participants or others? 

- Planned behaviors. If participants cite a number of intentions of ^ 
change in the future., additional follow-ups may be required. \ 

-Non-behavioral changes. Attitudlnal and' knowledge changes can he 
separated from behavioral changes in the data analysis, and each 
. . type of "change" can be "compared to course intentions. 
, -Course comments. Participants can be asked about their reactions to 
the dourse, from the perspective of having been back on the Job for 
a period of time. ^ ' 

ne^elooment of report - The data collected, analyses made, and conclusions 
drawn- can be presented in a report in whatever format avA level of detail- is 
appropriate to address the client's interests and needs. The results of a 



PAPA evaluation can be used for such purposes as: 

- Revising the training. *o 

- Attracting appropriate participants to the course. 

- Investigating possible suppj jinents to the training, such as technical 
assistance on the job, additional training, or job aids.' 

- Justifying continuation of the training. 

- Comparing effectiveness of different training programs or formats^ 

- Providing line managers with information they need to make decisions 
about sending employees to a training course. 

Following Up Without Action Planning 

A major variation on the PAPA process described above is to follow up with 
participants without having then fill but action plans during the course. 
Instead of starting the interview or questionnaire with action plan items ^ the 
follow-up can begin with a genieral question about anything participants may be 
doing differently on their jobs as a result of traihlng« Specific questions 
and probes can follow, as with the standard approach. If the course content 
can be segmented by objective, module, sub/ject matter, etc., specific 

. ' _ / _ ^\ _ _ ! _ 

questions can be added about any changes which ra^y have occurred in each 

J _ _ _V ' . > . 

specific area* As with PAPA, concrete examples^' of behavioral change are 

critical to data quality. J 
Agency Examples- o£-PA PA Applications 

During 0PM *s testing,^ of PAPA, the method was used in the following courses: A 
management communications course (using transactional analysis) for civilians 
in a military settinjf:; a personnel management course for supervisors at a 
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large agency; ah Interagency pdsitlbh tnanagemeht course for persbhnelists and 
supervisors; a coarse on training evaiaation previously offers! by 0PM; a 
team-building coarse for managers in two' divisions ^t a military-Jjase; a 
university-provided course for local police managers. The PAPA applications 
sammarized below were done independently by the a^^encies cited and demonstrate 
various uses of the method. 

Department of Energy - The "Participant Prepared Action Plan" was incorporated 
into DOE '3 Management DevelopnHEtnt Seminars This five-day coarse taught by 
contractors is attended by mid-level managers^ The course covers a range of 
basic management techniques, particularly emphasizing decision-making, • 
problem-solving, team-buildiug, and interpersonal communications : 

A standard action plan process was used to evaluate the seminar. Care was 
taken at the beginning of the class to explain the action planning and follow- 
up procedures. Participants were given guidance materials and forms early in 
the session^ with examples of action items drawn from the TX)E setting. 

A member of DOE's training staff conducted fcllbw-up interviews. Results 
cdnfirmied that the iseminar was essentially on target, but that it could oe 
improved in a couple of ways, namely, that certain areas .should receive more 
emphasis in the course than others, and that more opportunity should be made 
available during :he course for application and practice of certain skills. 
These suggestions were provided to the contractor, with changes to be made in 
future offerings^ — -■ 

The contractor was supportive of the action plan process, as the process 



128 

tended to^create more incentives for the participants to concentrate on how 
the c'ourse information could transfer to their jobs. As has been shown in 
.other action plan efforts, participants responded positively to the notion 
that the training office wis interested in how the participants were 
transferring what they learned to their Jobs. Participants seemed to 
appreciate a reminder after a period of time that a training staff member 
would be contacting them for a discussion of action plan implement atioti. Thiis 
the action planning procedure served as a vehicle for motivating participants 
to change, as well as a means of assessing change. 

For information on IX)E's use of action planning, contact: 
Stephen Rosen 

Employee Development and Training Division 
Office of Personnel^ Room 4E040 
U.S. Department of Energy 
Forres tal Building 
Washington, DC 20585 

(Phone; (202) 252^84%) \ 

Test- and-^Evaluat ion Command^ Department of Army - The Test and Evaluation 
Command (TECOM), part of n\RCOM^ Department of the Array, has incorporated the 
requirement for' an action planning process based on PAPA into a regulation on 
evaluating civilian training effectiveness. The regulation was deveic/ed as a 
-means of relating training to Job performance and raission accorapiishinent , and 
covers TECOM headquarters and its installations (including, for ^exaraple, 
Aberdeen Proving Ground and White Sarrfs Missile Range). Before being issued^, 
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the process .3 tested at a headquarters division, in order to ensure that the 
syste^i was practical and provided useful information. 

Under the regulation, installation civilian personnel offices send out a form 
(Supervisory Obiectives and Employee Action Plan) to supervisors of employees 
scheduled to attend certain trainin^i The form requests that supervisors 
describe their obiectives in nominating the employees for the course and what 
is expected of the eiiployees as . a result of their attending the training. The 

supervisors are to_ complete. that first part of the form at least one week 

before the class begins and are to provide the fonr to the nomi'.ees; 
discussion between supervisor and employee is encouraged. Within five days of 
course completion the employees are to complete the second part of the form^ 
which includes an action plan sheet along with guidance on writing action 
items (adapted from PAPA). The employees are then to send the original action 
plan to the civilian personnel office of the Installation^ with employee and 
supervisor each retaining a copy of Ithe plan. 

Three to four months following completion of training^ the personnel office 
sends a second form (Evaluation of Supervisory Obiectives and Employee Action 
Plan) to the supervisors. ' The employee flllr out the first part of that form, 
answering the following questions: 

1. Describe how you tried to implement your action items (what you did, 
how you did It^ who or what was involved). 

2. How is what you are doing now different from what you were doing 
before you attended this course? 

3. What; do you perceive as the benefits to TECOM, your organization, 
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ot to ybuf job performance? (Examples: dollar savings ^ reduced 
time, improved methods, increased productivity, better finished 
prodact, etc.) 

'4* If you were unable to carry out any of the action items, why? 
(Has the right opportunity not presented itself? Have things 

changed in the organization so that certain items are no longer 

_ „L ^ 

relevant-^? Are there other reasons?) 

-/ 

The supervisor uses the second part of the form to answer these questions: 

ii To what extent were your objectives met? (Check one: Fully; 

partially; none)i If you checked ''None," please explain whyi 
2i What is employee doing differently as a result of the train- 

ing? Give specific examples. 

3, What do you perceive as the benefits of the training to TECOM or 
to y6ur organization? (Examples: dollar savings, reduced time, 
improved methods, increased productivity, better finished 
product i etc.) 

4. Would you lecommend this course to others? (If so, who or what 
types of employees?) 

The follbw-up forms are returned to the installation's personnel office for 
analysis. Based on a composite evaluation of individual course analyses, the 
installation can assess the overall results of training cdrahpleted in relation 
to job/mission improvement. Feedback can be provided to managers and 
supervisors on the results. Individual personnel offices may choose to obtain 
additional or expanded information through such means as interviewing 
supervisors of part iciparit isi 

l4y 
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This process Is required for only certain types of civilian tfalnin^^ namely, 
those of 4b hoars or more, re^jardless of cost, and those costing S500 or more 
for a combination of taitton/fee, travel^ and per^diera. Exceptions to these 
requirements are courses vjhich are part of a planned^ progressive career 
development program (e.g., intern, executive development, upward ' mobility 
programs). These programs all include individual development plans and 
continual monitoring 6f employee progress^ 

The installations were asked to report on their experience with the system at 
the erai of the first year of operation. One general conclusion is that the 
system has caused supervisors to think carefully about the link between the ^ 
training being considered artd its contribution to the eraployiee's job 
performance. Some installatipns were able to show* training-related dollar 
savings, even though much of the training in TEHOM consist of highly 
technical, state-of-the-art courses for scientists and engineers. 

For information on TBCOM's use of action planning, contact: 
Derartment of the Army 

Headquarters^ U.S. Army Test and Evaluation Command 

Aberdeen Proving Ground, MD 21005 

ATTN : DRSTE-PT-C 

(Peggy Harmon) 

(Phone: (301) 278-4170) 

r 
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ANALYSIS OF THRJEVALOftTIQW METHOD 



- Willingness of course director to have action plan process built into 
courses 

- Or^anizationai support for follow-up contact with, participants after 
course completions 

" Instructions and forms for In-course action planning and for following 
up (interview formats or quest ionni ares )• 

- Participant "buy-in" to action plan process, and cooperation during 
follow^p data collection. 

Resources required 

The time required on the part of the evaluator is primarily related to 
conducting: the action plan exercise during the course - about an hour 
(although this could be done by the instructor); preparing for and carryir^g 
but the follow-nip (interviews may take 30 - 60 minutes each); and analyzing 
and writing up results (can be double or triple the time that it took to do 
the intierviews if that option is usied). Clerical staff can be uised to prepare 
questionnaires for mailing and tracking their return. ^The participantis , aside 
from ispending time to do the action plan, are requiried to devote thie 30 
tninutes or so to . ' interview or questionnaire process. As PAPA is a generic 
evaluation method, the standard forms, instructions, interview guides, etc., 
can be adapted to any particular situation with minimal resource investment.^ 



Two kinds of expertise are required to apply PAPAi If the recommended follow- 
up approach is chosen, interviewing skills are needed i Training can be 
provided to those who conduct the interviews; also time can be saved by using 
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more than one interviewer^ as long as all inteiTriewers have the same 
understanding of tte nature of the course, the questions to ask^ and the type 
of information which is to be -ollected. Basic analytical ability is needed 
to sort and synthesize the data and draw logical conclusions, but no 
statistical analysis is required • 

Nature of— i4vfariaat4.op-pr oduced^ 

PAPA yields qualitative data primarily on behavioral' change brought about by 
training. Tte changes may have been those intended by the instructional, 
process or they may have been incidental to the course's objectives (such as 
infortnat ion obtained from others In the class and useful in "the participant's 
own setting). Depending on the follow-up questions asked* information may 
also be obtained on participant reactions to the course, on what they felt 
they learned, and on the organizational results of any behavioral changes. 

----- a <• 

Adva^ages 

« _ . 

- Can be used to assess a variety of courses, particularly those difficult 

to measure through objective techniques. 

- Can be used by Itself, or In concert with other evaluation approaches. 

- Design of the process, and nature of questions asked in foiiow-up, can 
be changed to meet particular information needs and organizational 
circumstances, while retaining the basic approach. 

- Does not require extensive evaluation or tneasurement expert ise* 

- Can fie used for differerit training and development formats, e.g., short 
classroom courses; executive development programs (individual develop- 
ment plans can become the action plans; follow-ups can be made after 
each Assignment or course); career counseling efforts (again, an indi- 
viduai development plan can be the vehicle for follow-up by the 
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counselor); 

- Course being evaluated need not have behavioral obi ectlves ^ nor are pre- 
course measure necessary to assess change; PAPA can still be used if 
either or both of those conditions exist, 

- the action plan can serve as a vehicle for supervisor/participant 
discussions about using what was learned on the job, 

Ltini t at ions 

7 Relies on self-reports, and on skilled interviewers to obtain specific 
information needed* 

- Time required for interviewing and analysis; possibility of limited 
quantity and quality of data if questionnaires used; 

- While PAPA can provide a "red flag" for areas needing iTuprovement , l 
additional data should be gathered if consideration is being made to 
discontinue course. 

- Is not the best approach to use when more direct means are available 
to assess behavioral change. 

Reference 

Ruth Salinger \ \ 

Training Resources Management Hi vision 
Office of Training 

U.S. Office of Personnel Management 
Box 7230 

Washington^ T5C 20044 
(Phone: (202)653-6173) 

14 i 

o .. . ■ 
ERIC 



ATTACHMENT 1 



GUIDELINES FOR WRITING ACTION ITEMS 
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)EA1 FOR ACTION ITEMS i 



iirse bates \ 

ft _-- . 

i!3 I would like to try out when I return to work, based on what I learned in this course. 



I 

14 



ote: You can use the course objectives, what you learn In class, the course handouts, conver- 
itlbhs with others, etc.. to come up with ideas. 
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GUIDELINES FOR WRITING AGTION ITEMS 



What an Action Item Looks Like 

A, The most Jmporjarit cha 

action item Is that it is jvritten so that 
you-^pr sbmeqne else-^WLlJ Know when ft 
occurs. One way to help achieve this is to 
yse specjfjc acf/pn ye^^ The following is 
a list of such verbs: ' 



my uhit. and cdrtie to ah agreement on 
whether or not I can proceed. ; 



II. ImpJementInQ the Action item 

A. As yoD^proceed to develop action items, 
be_s_u_re to f/j/n/c of yourself: in your actual 
lot setting, implementinb the activity you 
have described. 











4M«ntal Skill 


Physical^kill 


Attitude 


State 


Demonstrate 


Execute 


Choose 


Name 


Discriminate 


Operate 


Volunteer 


Describe 


QIassify 


Repair 


Allow 


Relate 


Generate (a solution) 


Adjust 


Recommend 


Tell 


Apply (a rule) 


Manipulate 


Defend 


Write 


Solve 


Handle 


Endorse 


Express 


Derive 


Manufacture 


Cooperate 


Recount 


Prove 


CaMbrate 


Accept 




Analyze 


Remove ^ 


Decide to 




Evaluate 


Replace 


Agree 



B. As you are working Oh the actibh items, 
ask yourself: /5 //7e it)e/3av/of described 
odserva/)/e? Will it be obvious to me or 
others when it happens? 

C: Examples bf action items: 

As a result of being in this course I plan 
to: 

1. Describe this course to my superyjsor 
within a week of my returning to the 
job. As a result, my supervisor will — 
know: the contents of the course; how I 
ca^n^aAplyJyhat _l_l_ea_rne_d_t_o_ the job; and 
whether or not others in the organiza- 
tion should attend. 

2: Handle 'every piece of paper only once 
in order to improve the management of 
my own time. Begin as soon as ) am 
back on the job. 

3, AppJy the principles of performahce 
analysis to the probiem of incomplete 
and tardy case reviews In rny division; 
request assistance from the training 
office^ as needed. As a result, I will 
know whethe^r training is required 
and/or some other solytlpn is 
appropriate to reduce the problem. 
Begin within a month upon returning. 

4, Talk with my employees directly about 
a problem which arises^ rather tjaan _ 
avoiding a confrontation; discuss the 
isltuatioh in order to reach mutual 

understanding. ' 

5, Within two weeks after I return/* 

negotiate with my supervisors to irppie- 
merit a \ systerri in 

* Ch«rt rnodified from rni0rs9mc» Proce^^^ for mimcmni) 
Sysf0m$ ^^fopm&nt: Phsse JlDtsigh, Rd6ei1 Branson, et: 
O Flbridl State University. Ai>A019. pace 12. 
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y_yb u have ah jdea when you _w [ n_be 
able to begin Implementing the action 
Lteths, ypu can rhake a note of Three 
categories can be chosen; (1) "as arises" 
Jydu ddn*t know when the dppdrjuhity to 
try thjs Itern will occur);^2) **wjthin 2 
ffldnths"; and (3) "after 2 rndhths," 
You may find that you cannot try out youi 
ideas exactly as you envisidned them, dr 
that it is difficult \6 be specific. That's 
o.k.— it is still »mpdrtant td write dut ydur 
intent, as a tentative plan, knowing you 
may have to motfify it once you are back 
dri the job. Try to develop at least two or 
ttiiee action items — one may not work, so 
Ifs handy to have others. 



14 



138 



AGTiON PLAN FOFM AND QUESTIC^^S 
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ACTION PLAN 
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COURSE TITLE 



NAME 



DATES 



ACTION ITEMS 

i plan to: 



ERIC 



_ Within 
2 ftioftths 



14 



_ after. 



_ as _ 
arises 
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QUESTiONS ABOUT YOUR ACTION ITEMS 



A. Preliminary Nature of Plan 

Were you specific in writing the actloh item? 
What wili you need to do when you returr^ to 
' work in order to find out which actions are 
possible? 

B. Resources 

Who would be carrying out the proposed 
action, or helping with It (formaliy or 
Irtformally)? 

Are the skills for carrying it out available? 

How much time would this take? 

Are there special materials or equipment 
• required? 

What is inyolyedJh pbtalrijng them? 

Will you be using tool or system or aid 
ftom t hjs iojj rse? 

How much adaptation Is required? 

Is iLOrtinual monitOTing or follow'through 

.required? _ 

Who will do it? 

C l^mpiementaUon 

Do you h^ve the authority to implement the 

actidri? 
If not. who does? 

HoKi do you think you can go about getting 
approval? 

What do you think the degree of support is 

fk.r your idea? 
Win you need to sell people on it? 
Who? 

D. Effects __ 

Whom will this action affect ? 

How will it affect them? 

Will* anyone be the worse for the results? 

Anyone Improved? 

What will be affected? 

E. Environment 

What in the organ izatiqhai environment might 

Interfere with year doing this? 
What in the organization would support your 
. efforts? 
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ATTACHMENT 3 



FOLLCW-UP LETTER (INTERVIEW) 
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SAMPLE FOLLOW-UP LETTER 



Dear 



Several rhonths ago you attended the course ( narhe of course ) . At 
the end of the prbgrarri, you developed some action Items—- ideas for new 
behaviors— you might try out once you returned to your job. At that time, I 
said that I would be getting in touch with you to find om how you were 
doing in Irnplemehtihg your plans. 

Now is the time! Either I or another rherhber of otir staff will be calling you 
In a week or so to rnake ah appointment to taiR (by tejephonej about ypur 
action plan. As a result of our discussions wl.th all the course participahts, 
we will be able to tell the course director h.bw the course has affected the 
way participahts now do their jobs. As you know, our report will cover the 
class as a whole, and no names will be used. 

«»• 

During bur ebhversatibh we will ask ybU hbw ybu fared with the action iterns 
ybti wrbte dbwn, about anything else you've tried since returnjhg to your jbb, 
and about any problerns ybu might have encountered. This Ihfbrrhatibn will 
be used tb irn prove ihe course and ensure its usefulness to participants 
once they are back on the job. 



Sincerely yours. 



INTERVIEW FORMAT 
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INTERVIEW GLJiDE AND DATA LOG 

Directions 

This attachment contains suggested questions to be askeo during the in- 
terview. Probes for getting very specific ihforrnatibn follbw the ihitiai 
questions. Use therh as appropriate. Space is provided below the ques- 
tions for recording the participants' answers. 

Gburse today's Date - 

Does participant have action plan? jntervlewer 

Name 

Guide for the Interviewer | 

This is ( your name ) from ( name of your office ) . \ 
We talked ( say when — — ) and arranged to discuss your action plan. \ 
Is this still a good time? \ 
Do you have your action plan from the ( name of course j course 
In front of you? (IF "NO") Can you get It now? (IF ''NO") We can work 
frbrh rhy copy. • ■ \ 

What I'd like to do is gb thrbtjgh the iterhs you wrote down at the end of 
that course, arid ask you some questioris about them. Then you can bring 
up anything we might have missed, including any behaviors resulting 
frorh training which were not oh your action plan. 

(Make sure answers are specific enough so that you can envision the 
behavior being described.) 

Let's start with tine action item which says, - - — , - -- -- 
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INTERVIEW GUIDE AND DATA LOG 

Have you been able to do this yet? 

1. ( IF ••YES," THEN USE THESE STANDAfRD PROBES . AS 
APPROPRIATE. IF ••NO"— GO TO # 2.) 

• Could you tell me more about that? What was the result? 

• Could you give me an example of that? What was the result? 

• How wouJd you characterize the result? Was It positive? _ 
Negative? Did It have both positive ahd hegati^^^^^ 

• How did you carry that but? Who was involved? Have they been to the course? What 
was the result? ? 

• Have you done that more than once? Any difference each time? 

• How IS that different from before? 

• Were there any problems in carrying that out? What were they? 

• Will you continue to do this? Foresee any prbblerhs? 



Record answers below: 
WHAT (with exarnples): 



RESULT: 



HOW GHARAGTERI2ED (positive, negative, mixed, unknown): 



BEFORE: 



PROBLEMS: 
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INTERVIEW GUIDE AND DATA LOG 

2. (IF ••NO/* THEN USE THESE PROBES, AS APPROPRIATE.) 

• Why do you think that was the case? 

• Any other reason you can think of? 

• If the (problem) had not occurred, do you think you could have done that? 

• Are you still interested In doing this? 

• If so, how do you think you can make it happen? Foresee any problem? 

Record answers below: 
WHY NOT: 



OTHER REASONS: 



STILL INTERESTED: 



CAN YOU MAKE IT HAPPEN: 



(If apprbprlate, use previously developed questions to relate behaviors to 
learning objectives.) 
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INTERVIEW GUIDE AND DATA LOG 

3. O.K., we've been talking about ^ 



Have you done ahythihg (else) similar to this? (If •TES/' use standard 
probes listed under #1.) 



(Repeat questions #1-3 for each action item.) 

4. We've talked about your specific action Items. Is there anything else 

that you are doing differently oh your job since attending the course ^hat you 
think is due, directly or indirectly, to your being in that course? (If ••YES," 
use standard probes listed under #1.) 



Ahythihg else? 



5. Is there any way you think the course should be changed, to make it 
more useful to you on your job? 



Anything else? 



6. (Optional) What do you think about writing an action plan at the end of 
the course? Did it help you apply what you'd learned? 



7. (Optiohal) What do you think about this follow-up? 



is. 
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OUESTIOWNAIRE FORMAT 
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SAMPLE FOLLOW-UP COVER LETTER 



Dear 



Several months ago you attended (name of course) . At the end of the pro- 
gram you developed some action items— Ideas for new behaviors — you 
rnight try but after you returned to yburjbb. At that tirhe, I said that I would 
be getting in touch with you to find out how you were doing in implementing 
your plan, 

r^ow is the time! Enclosed are a questionnaire arid a preaddressed envelope. 
For our follow-up evaluation of this course, we would like you to answer 
some questions about your action Iterns and the usefulness of the course to 
yeor job. Directions for eompletirig the quetstidrinaire are enclosed. If you 
have misplaced your action plan, call me at (your phone number) and 1 will 
send you a copy. 

As ybu were told in class, you can be sure that your privacy wilF be protected 
In any use made of this questionnaire. Our report will cover the class as a 
whole, and ho names will be used. 



After cbnipleting.the questionnaire, you may want to let it sit a day or two 
and then reread it before sending it directly to (address) in ^^e preaddressed 
envelope. We will call you to remind ybu about returning it i ; if we have 
not heard from you. by ( date ). Thank you for your cooperatic s 



Sincerely yburs» 



Enclosures 
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QUESTIONNAIRE 

Directidhs For Filling Out the Questionnaiire 

the questionnaire contains three sets of questions: X, Y, and 2. 

Set X should be used for action items which you have tried to carry out, 
regardless of how successful you were in your efforts. 

Set Y should be used for action items which you have not yet tried to carry 
but. 

Note: For any action item, use only one set of questions, either X or Y^ not 
botli 

Set Z should be answered regardless of what happened with the action 
Iterhs. 

Procedure 

1. took at your action plan and, taking each action item in turn, decide 
whether it is a Set X or Set Y item. 

2. Copy each action itern In the blank at the top of the appropriate set of 
questions, either X or Y. 

3. Write youE ansv/ers directly below the questions. If you run out of space, 
use the other side. Any unused parts of the questionnaire can be thrown 
away. 

4. Set Z eoritalris questions that are self-explanatory. As before, please 
record your answers below the questions: 

5. After you have completed your questionnaire, please plape It In the 
enclosed envelope and mail it back as soon a5> possible. 

thank you. 
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SET X 

Name — . — ;-- - Date : -'— 

I am discussing the action Item which says: . 



a. besoribe how you tried to Implement this item, Including, as appropriate, 
the followlhg information: 

• What you did or said 

• How you did it , 

• Who was involved 1^ 
^ How often you tried It - ' | 

Please be specific enough so that the reader can visualize what happened. 



b. Describe what happened when you tried to implement this item. 

• What happened as a result? \^ 

• How do you feel about what happened? 

(Was it pbsitive, negativ/e, partly positive arid partly negative, or don't you 
know yet?) \ 
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SET X 

c. How IS what you are'doihg oh this action Item different from what yoti 
were doing before you attended the course? 



d. Did you have any problems in trying out this action item? If so, what or 
who got in the way of doing it? 



• VVas there anything else you needcu to know in order to carry out this 
action item? / 



e. Do you think you will continue to try out this action? 
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S ET Y 

Name E)ate : — ^ 

I am discussing the action item which says: 



a. Describe why you vwere hot able to carry out this Item yet. For example: 

• Has the right opportunity not presented Itself? 

• Have things In the organization changed such that it is ho longer rele- 
vant? 

• Do you heed other information or skills to try It? (What are they?) 



b. Can the action Item still be done? If'so, what is needed to rhake it 
. happen? - . 
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SET Y 



c. Are you still Interested? if "NO',' why not? 
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SETZ 

Name — 

a. Is there anything else that you are doing differently on your jejt) that you 
think is due, directly or indirectly, to your having taken the course? (If so, 
please answer any of the questions In Set X which apply.) 

/ 

/' 

/ 

b. Is there any way you think the course should be rriodified to make it more 
useful to you on your job? Pleasfe list any suggestions for improving the 
course (course content, structure^ length, etc.) you rnlght have. 
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c. {Optional) Do you feel that developing an actiofi^plan as part of the course 
was a helpful exercise? Explain why or why not. 



\ 

d. (Optional) Did you find this follow-ti^ questionnaire a helpful exercise? 
Explain why, how, or why not. 
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Appendix A - Overviews of Training Evaluation 



Appendix B - Selected Bibliography 



Appendix C - U.S. Office of Personnel ^uinagfenent Contacts 
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NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
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(Excerpts) 

IMPACT OF TRAINING IfLESft 
1978 -1980 



To assess the_ impact of training on the falfillment of ESA's mission itis 
fiecessary to lobfe. first at What contributions the various training initiatives 
have made to achieving E£A*s shbrt-ahd long-term objectives. Having catalogued 
or listed such eohtributi .is^ we must then examine the training itself, to see 
if it was done effietehtl:'. By examining both these areas, we will be able to 
evaluate the cost-effectiveness of ESA's training efforts. 

the first task in such ah approach is to identify ESA^s basic objectives. While 
it is possible to eohceptual ize the organization's ultimate purposes in a number 
of ways, for the pnpose of this report We will consider three: 

1 Achieving effective operations (production). 

Providing public service. 

3. Developing effective managemeRt. 

Biveh that most or nearly all ultimate benefits to the organization will reduce 
themselves to one of these_ L^^re r?tegories, we must then attempt to examine how. 
training has contributed to them, ^ince th.-se are broad categories ^ it is useful 
to subdivide them into more spec^''^: Clemen' s that may be easier to define and 
identify. Following is our. a tternK. to make that division. Note that this is not 
an attempt to be comprehensive. However, the particular items enumerated are 
areas central to ESA's goals in which training might conceivably make a direct 
contribution. 

! 

Ac hieving Xff<?ctiye 4)perjtions^ 

1. Achievin g Prpgram -Eoe^^ - Each program and sub-program sets a number of 
quantitative goals by which performance and production are at least . 
partially measured. 

2. Qualit^^ Kor t ^rodu^ts - The work>/h1cfi is measured quantitatively 
(i.e. production goals are set) must iilso meet standards of accuracy and 
completeness in order to be effective. 



Providing Public Service 



3. Quality of Service - Factors, important to serving the public include 
providing accurate and intelligent information to individuals and groups 
when required^ as well as making and implementing accurate and timely 
decisions affecting the public. 

4. Rapid Service - Not infrequehtlyi the. val ue jof a service is heavily 
dependent on the timeliness of its delivery. Reducing the time lapse 
between the passage of hew leglslatiohi or acquisition of staff and their 
becbmihg productive is a significant staff cahtributibh. 
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Bevel bplhq Effective Ranagement 

5. Efficiency of Operation - Efficiency here refers to accomplishing a job 
more rapidly, more simply* With low errors » and with high quality. 

6. Fiscal Respohsibility - Because of the critically high importance of 
responsible money management in government, adherence to required accounting 
procedures, and accuracy of records are crucial. 

7. Increased Management Capability • Due to Increased demands on managers for 
the mastery of a broad range of management tpolSt including the use of MIS, 
demonstrated improvement In this capacity Is a direct contribution, 



Cost-Efficie^^ey In^aiiiing 



.«&^^urther information Is needed to determine the efficiency 
and economy of ESA's training efforts. This analysis should Include an 
assessment of both the cost benefits of ESA's centralized training function 
find the cost efficiency with which each course Is designed and delivered. 

The fact that ESA's training function Is centralized provides the primary 
opportanitiy foi* cost-erficisncy In the entire training effort. Savings 
result frcn several factors: 

— Irc-lning resources Ibeatsd ir, the centralized unit can be quickly 

moved from program to program as organizational priorities Shift. For example, 
resources can be oaickl^ diverted to areas where there is new legislation, a 
consolidation, or s ^Ajor change in erg<ihizational procedures. 

-= Because It possesses a flexible cadre of Highly trained learning 
specialists, ESA is rarely dependent en. the use of outside cbhtractprs for 
training. When such contractors are used, they always supplement skills 
.^lat already exist on the staff rather than providing skills hot available 
Internally. This means that central staff can inonitor the quality of work 
p,. formed by contractors as well as limit its use. 

The fact that the training staff becomes familiar with subject 
matter and skills within various ESA programs means that they can often 
apply successful methods and techniques developed in one program to another 
program in ESA. This avoids waste and duplication and provides a cross- 
fertilization of working ideas. 

To analyze cost-efficiency in specific courses, we. need to examine CHrrent* 
research and practice in the traijiing and development field. Such knowledg : 
Indicates that certain chara.r eristics are more- likely thap others to insure 
efficiency and economy.^ These are briefly listed below: 
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Focus on Specific Job Skills 

--Courses should focus oh specific, identified job skills that 
trainees heed to know in order to do their jobs. This approach, which 
eliminates unessential material , keeps courses bri and tightly focused 
on performance artas as possible. 

Written ResourceJaterials 

--Course should include written resource materials that trainees can 
take back to their offices and use^^as ongoing job aids. 

; • 

Decentralized Delivery 

--Courses should be delivered in decentralized locations so as to 
reduce travel and per diem costs, which are the highest single costs for 
delivering training. 

Self Instruction 

Self-instroction, in which trainees work through prdgramned learning 
materials, is a very economical training method, particularly when the 
learning involvesa transfer of knowledge rather than skills. ESA courses 
utilize, self-instructional materials wrtenever feasible. 
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INTRODUCTION. 

To many in the f-ield of training, evaluation seems, too complicated, too^ n or too 
costly to consider^ or it warrants only limited commitment - just enough to satisfy 
minimal needs to appraise pfogf^m success or satisfy accour lability requirements. 
However, the gains to be achieved through cost-effective, reliable, and objective 
evaluation of training are substantial. It is the purpose of this report to highlight these 
beneiits and to demonstrate that they are within the reach of every training organization 
within NASA. 

We will attempt .o do this by offering a typology and termiinolgy for understanding, 
analyzing,, and developing evaluation strategies based upon a review of the 
state-of-the-art of traininjg evaluation. The topics to be covered include identification 
and de.finition of the objectives of evaluation, a summary of the major concerns and 
factors to be considered in evaluation, a review of techniques and instruments to match 
with objectives and*^ issues . 

A strictly pragmatic approach is used. Our purpose is to present a point of view ^and a 
way for the "average'* training office to use evaluation in order to benefit the employee, 
the organization, and the prospects of the training community. While our intent is to 
identify and desc: key points and factors involved in training evaluation, we do not 
attempt to proscribe a specific measurement technique or approach. We believe the 
needs and circumstances of each particular situation will largely determine the choices of 
evaluation objectives, techniques, and approaches that are appropriate. 

BENEFITS OF EVALUATION 

The potential benefits of training evaluation are extensive. Evaluation can be the 
initiator of improvement in several key areas, i.e. the training course or program itself, 
the situation of the training office, the credibility of the training manager, and the 
opportunities of employees who utilize the service (Zenger and Hargis). These benefits 
are discussed more fully below. 

Impro vement of the Training Office Situation. Accountability poses a problem for any 
3?gantzation, including .the training office^ Collecting and presenting pertinent evidence 
of the need for training is a constant requirement. Management support often_i_s_ based 
solely on impression and temporary favor which is highly susceptible to change. Effective 
and effectively used evaluation provides a constant source of data to account for the 
resources applied by training offices. Clearly, a danger is apparent in those instances 
where results are deficient e.g. training is poorly received, learning objectives are^not 
achieved. But evaluation data utilized properly to redesign and reprogram will produce 
beneficial results. : 

Bene fit to the TraininR-^riaRv^ . Training managers can utilize evaluation in two 
important ways. First, they can uL*e evaluative responses to make changes in instructional 
techniques, program management, and course design. And second, they can improve their 
credibinty with management ' through the use of assessment data. Evaluation dAta_ can 
help strengthen management's confidence in th^ training specialist as nnanagement 
realizes that the training professional shares their concern for performance improvement. 
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Benefits to the Client . The prime commitment of training is to the. client, i.e. the 
trainee and the trainee's management sponsor. Ultimateiyi, the goal of . evaluatibri arid 
programmatic improvement based on evaluation is the enhancement of empilbyee and 
organizatior^al performance. , As improved training results in better job performance or 
enhanced career ppportuhlties, both the emplbyee and the organization move ahead. 



Trainir 



i*** ^ ! " a nd 



Evaluation provides feedback needed to 
yPPr^Y? ^J^^tJl ifistructional content and methdd^^^ While it is possible to design arid 
conduct trainin^l^ based on a combination of content familiarity, instructional expertise 
and trainin^g sophistication, there are difficulties with this approach. Intuitive judgment, 
even based on co^nsjderable experience and insight^^ PrP^i^? consister[t^ 

objective assessment required to appraise the large number and wide variety of training 
inttatives offered by the contemporary training office. _ The danger in using subjective 
evaluation of courses and programs is in the resultant tendency to leap to misleading 
assumptions. These assumptions serve to' shorten the program development process as 
showri below (Goodell, p. 263): 



Needs 




Objectives 


Identified 


— ^ 


Defined 



Trairiirig 

Course 

Designed 



Noticing participant 
response during ; 
course 



Assumptiori that the riionitoring 
of signals is a substitute for 
a post-course evaluatiori 



But the "shortened process," relying solely on the impressions, ^opinions, and views 
collected \yhile monitoring corriments and other signals, negates the ability to effectively 
r^^?f^:n"^_^^jectives and methods. Includ^^^^ in the program development 

process gives, a more reliable basis for revision and redesign of the training: 



Needs 
identified 



Objectives 

Def i ned 

vfv 



Training 
Course 
Designed 
(or re- 
designed! 



Monitoring of 
feedback during 
course 



Revision of 


^ ~ 


Objectives 





Post-course Evaluatibris 

(formal & informa 



EVALUATION OBJECTIVES 



bur typology of evaluation objectives or data targets includes four basic types: reaction, 
achievement, performance/apj3lication, and organizational results. These are describe J 
in more detail below. (Kirpatrick, p. 1; Zenger and Hargis, pp. 11-12; USC5C, "A Process 
for the Evaluation of Training", p. 10) 
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Reaction 



The objective of this. type of evaluation is to assess part^cipartr* rrricrions to .'f-u.avks 
based upon the training experience- Reaction evaiuetion :)btajning dat:; on traipeeV 
feelings and opinions about a program, yields infa^'Tia l about instr jr o i, co'^teriii 
methods, media,, and program brganizatidh* The most common techrj-fj - r.;.- ; ^ed. to 
gauge reaction aVe questionnaires, oral review sessions, and indiv:du^>i Lr . ' vvt.-?. Key 
anecdotes and testimonials may be . collected. It is important to_ pom: . that 
measurements of tr^ainees' reactions do not include the assessment of achievernent of 
learning objectives. , 

Achievement 

The objective of achievement evaluation is to determine trainee skill or M.owledge gains 
\yhich reflect the accomplishment of instructional objectives. Achievement 
measurements gauge the degree to whkh concepts, principles^ facts* skills, and/or 
attitudes are acquired by partici^pant5 ' the training. Achievement measurement 
devices typically include direct observation, written or perfbrmahce tests, and 
self-assessment processes. These devices are intended to measure end-of-course learning 
achievement and. do not predict the degree of transferability of newly learned abilities to 
on-the-job situations. 

Performance/ Application 

The objective here is to determine .Ghanges in individual performance on the job which are 
a result of training. Performance and application measurements entail examination ol 
changes in individual employee job behavior or practices attributable to training. 
Interviews^ observation, performance tests, questionnaires, self -assessment processes^ 
and performance appraisals are commonly employed -to determine performance changes 
due to a learning/training experience. The identification of specific work place activities 
and functions which training is expected to influence is an essential element-in this type 
of evaluation. . . _ / 

QrRanizationai Results ' 

The objective of this kind of evaluation js to determine the impact of training on an 
organization or the work enviionment. Examination and measurement of this impact 
usually involve analysis of such changes as costs, absenteeism, turnover rates, and 
organizational prDductivity. Such changes are commonly ascertained by the use of 
interviews, analysis -of organizational documents, questionnaires, and surveys. From a 
training evaluation perspective it would be ideal to evaluate^prdgrams directly in terms of 
the organizational results. Hcvever, this is often difficult because of the problem of 
seereeating variables to determine how much organizational improvement is ^ due. to 
trairring as opposed to other factors* However, inferences can be made about the effect 
of training on organizational improvement which substantiate the effectiveness of training. 

MAJOR EVALUATION CONCERNS ; , 

Having identified the objectives of evaluation, let us now look at the factors that must be 
considered in detej-mjning whether and how to proceed wLth an evaluation. These factors 
include RIGOR, RELEVANCE, ECONOMY, and STAFF CAPACITY. 
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RiBof refers to the reliability*. validity, and accuracy of measurernent techniques. It i§ 
fhl amount of control exerted by ah evaluation design and the subsequent accuracy tha^ 
can be attributed to the evalUor.ion results,^ The more rigorous the ^design, the greater the 
level of confidence in the evaluation results (Zenger and HargiSj pA2). 

Relevance means the degree to which 'the evaluation couples with achieveflhent goals 
^nd/or organizaJfonal goafs. It does not refer to the linkage between course content and 
gSals/buf to :he. degree to which the evaluation ^ results reflect the achievement of 
specified objectives. 

Econorny fefers to the trade-off between the costs and benefits of evaluating, A range of 
techSqUes ex St tha^ offers insight into the success or failure of .training. _Of ten a less 
expensive approach will provide one , or more of the benefits described in an earlier 
section Other techniques-:, require a substantial expenditure of Cai>itaj or human 
Sources to acccomplish a rigorous and relevant -evaluation. Two available options ar^ 1 
to offer limited and thoroughly evaluated training opportunities or 2) to offer a P^noP-Y 
muning that addresses management's needs, as they arise, without significant 
evtruation. The search for the^'middle ground" is one of the major questions confronting 
training managers (2enger and Hargis, pp.l2-13). 

Staff capacity is related to the issue of ecortomy; The availability of staff t^o conduct and 
inSyze Sation results is. a key concern. Obviously, the more _ sophisticated the 
meSt^ment, the more skill and knowledge is required. Evaluation _can range^ in 
?omSe<Sy from the ease of a single trainer soliciting a trainee's views in an informa 
co'v^Sation to Jhe complexity ol a large scale analysis of organizat^^ 
requiring a staff consisting of a training sReciahst, a statistician, and a- computer 
pXammer to determine kly variables and correlations. However, evaluation expense 
fs avai^b^ either tl.rough the development of the training staff evaluator or by contract 
fonsSta^on. (pJrVer, pp. 19-1 through,. 19-23: Parker provides an exceU'.n-t overview of ^ 
ihe methodological requirements for the various evaluation techniques.; 

jke Relationship of Concerns to Objec tives 

In practice, the concents described affect'the various evaluation objec|ves {i.e^^ HiS^^t 
acWevement, performance/application, and organziational results) in distincJy differ.^^^ 
wats The following section is a brie.f summary of how the issues of rigor relevance, 
economy, and staff capacity impact the four different evaluation objectives (Zenger and 
Hargis, [Dp. 12-13). ; 

' ■ ■% 

- \ . ' . 

Reaction 

Reaction queries af^ usually quite economical, tending to 'rely on brief ^questionnaires and 
Tntervlews'S rfdividuat trainees or groups. Reactive responses usua^b' lack ngor sA^^^^ 
D^rticioants Reneraily report likes or dislikes, anecdotes or incidents _that are not always 
di^Sfrelafe^tf the content or substance of the , training. Relevance of reactive 
TesroSes tS train ng obiectives is relati/ely high when training is conducted prin.^rily_ to. 

nature of reactions. . ■ - ; 
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Ach ievement " ' " . 

— : -J . _ . ^ ^ 

The measurement of learning achievement or knowledge gains tends to be more rigorous 
arid relevant than participant reactions- given that such instruments as written tests^ 
structured observatiorij, and participant yelf--asse's3rnents demonstrate directly XM level 
' of mastery attained. These- kinds t)f tests are relatively inexpensive and often may be 
easily developed by training office staff ^working 'with subject experts as necessary. In 
addition, they are appropriate for many arveas of training and are 'an^economical way of 
-achieving the benefits of .evaluation. In particular, they are useful in determining the 
^dequacy of the instructor ,a^ the course design. Also,* when achievement/ levels are 
^tighj the training office and the training manager can rQireive desirable management 
recognition. In addition, -evaluation of achievement when accorripaned by feedback to 
trainees reinforces jearnin^^result^^ For the training office which -takes time to develop 
the instruments, achievemem: testing has high potential payoff at relatively low cost. 



Performance/ Application 



^^easures of performance change and_application to the job may provide a goQd mix of 
rigpr^ relevance, and econpmy. . Such measures are rigorous in relation to their degree of 
sophistication and are relevant both to achievement objectives^and to organizational 
.needs- Although more sophisticated (and expensive) devices may be used, simple 
instruments such as qyestiojinaires can be constructed and adrninistered at relatively low 
cost. A potential problem arises from the necessity of constructing instruments which 
are able to evaluate on-the-job changes. Job or task .analyses are necessary in order to 
insure adequate evaluation. In these cases, personnel and evaluation experts will often be 
added to the training team^ thus adding to the cost. 

* 

Measurement of organiza tic- al results, i,e. organizational performance or produciivity 
changes as a result of training,^ is ob.viousiy relevant by* our \definition. In order to collect 
.the data requjr \ ^ highly rigorous design may be developed at ^some expense. Evaluation 
design and th ' :^retation of results may require the assistance of an evaluation or 

prganizationai -lopment expert. The -level of insight into the^mission and activity' of 
the organization must surpass that which is required for an evaluation of a performaYice 
,:Change of an individual trainee. The^^difficulties mentioned tend to make this evaluation 
effort a relatively expensive process requiring a high level of management support. 

* , EVALUATION, TiiCHNIQUES 

r.. ' - - - - 

Having decided which evaluation objectives suit need and circumstarice and the ability to 
•pursue them, we may now look at some of the techniques or instruments available to 
^ccornpUsh each objective. If is realized that the selection of instruments must be 
related to an overall evaluation design. These are included to assist training staffs to 
begin designing or developing their _own evaluations or to work more effectiv-ely with 
consultants and subject experts. For the .m:ost part, these techniques for training 
evaluation fall into five majb^ groupings: TESTS, GBSERVATIONS, INTERROGATIVE 
instruments, PARTICIPATORY techniques, anU 0_R_CLANIZATJONAL analyses. TESTS 
include, written examinations and 'simulations. OBSERVATIONS include ail visibly 
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ridticeable aspects of traihipg remits. INTERROGATIVE; Instruments include /such 
devices as questionnaires^ individual interviews^ and group interviews, v PARTIGIPATGRY 
instrurnents . include all self-assessment and participant _ contract devices. And 
ORGANIZATlbNAL analyses Include all measurements of an organization's vital 
documents and statistics. 



A definition and descriptibrv^ each individual technique follows. This section concludes 
with a chart summarizing the relationship of specific techniques to evaluation objectives 
(Chart I). 

Test s 

Written' Test this instrument consists of aseries df written items (questions, 'problems, 
exercises) which measure specific knowledge gains against specified criteria or 'norms. . 
The objedt of the written test is to ga^g^ the possession^ l^y^i.^l knowledges imparted to 
participants during training. Ihe information gained tends to be a ^elatlvelj' objective 
measure of cognitive and/or psychorn'oter skills; as such, written tests constitute an 
effective measure of AeHIEVEMENT. * 



Use Requirements: time, scoring procedures, and high correlation with training content. 

Advantages: - relatively low administration cost, easily and quickly scored, easily 
adrninistered, and a wide sampling possible. . . ' f 



Disadvantages: 3 possible low degree of relation to job performance, highly prescriptive^ 
and potential cultural bias. • ' 

Perform ance Test . The performance test is an instrument which requires trainee 
demonstration of learning (ACHIEVEMENT) or performance change 
{PERFORMANCE/APPLICATION) under s or actual cdndltLons. ■ The chief 

characteristics of data derived from j^erformance ;testing include quantif lability and' 
observability. Assuming sound design, this instrument offers great insight Into knowledge 
gain and actual application of learnings. 



Use Requrements: time, materials, equipment, and qualified raters. 
Advantages: reliability and relevance. ' ^ 

Disadvantages: liigh developmental costs, time-consuming, potential difficulty in 
constructing a simulation which mirrors actual conditipns, and possibly threatt^ning to 
participants. 

Observations 



Noil- ve rb al i n dic^ t i on This technique involves observation of gestures ahd__ facial 
expressioas among training participants; as such^ it represents a potentially effective 
vehicle to determine REACTION. The information gathered tends towar-d Mfective, ' 
^uncategorized, and Inferential physical indications of reaction to training. 

Use Requirements: sensitivity to physical. communication and evaluative awareness. 



Advantages: low cost and immediate feedback. * 
Disadvantages: subject to misinterpretation, low reiiability. 



EKLC 
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Ob<^>r-\.'iUion Ar. 31 .x. i.iuc technique, observation constitutes, organized surveillance of 
"n^Hrii^^par : ici pants: Evaluative observation. may be conducted either oh ah incidental or 
struc\:-^'^rf ; ;^i<;^ '^rrpending oh tlLe specific infcrrriatidh target, observation may provide 
asocrce •>v ^vLArr^ON, ACHIEVEMENT, arid PERFORMANCE/APPLICATION data. Jf 
vocai ^' . ■'■tAi.^ii indications are sought, reactive data are provided. If these indications 
substc. '.iMO » irning gain or compreherision, then achi,evem_ent is measured. And if these 
indicatir-'ns arc directly related to a change in behavior or practice, or are viewed 
actually on-the-job, then performance and application data are supplied. . 

If observation is conducted on an incidental^ basis by a human observer, the . data tends to 
be perception-based and randqm; if conducted by electronic means (e.g. videotape), the 
data becomes all-inciusiye and can be analyzed in totality or may be subsequently broken 
down. Structured surveillance requires specific observation (i.e. the observer is looking 
tor indications previously identified). 

Urvi Requirements: skilled observers and/or equipment^ . observational criteria, 
cooperation of training p^: ::cipants (and their supervisors if observation takes place in 
the work sett'ng). 

Advantages: human observation is usually low cost^ electronic observation provides a 
permanent record, if the observation method is concealed or relatively unobtrusive then 
the process is non-threatening. 

Disadvantages: unconcealed observation, either by human or electronic means, can be 
tfireatening; recording equipment can be expensive; concealed observation raises privacy 
issues. 

I .terr^ gative Instrume nts 

Ihterview An interview is a structured conversation in which an interviewer asks 
questions to obtain information from and jmpressions about the .training . participant or 
those associated with participant. The interview may seek REACTION data through 
solicLtatiori of opinions, judgments, or feelings about the training; attempt to gauge 
PERFORMANCE/ APPLICATION levels through questions Jregarcing behavior change or 
learning application; and/or seek to ascertain ORG ANIZATIONAL RESULTS through 
questions designed to identify change in the target organization due 0 t^'aining. 

Use Requirements: time, trained interviewers, training participant^ and/or organizational, 
cooperation. 

Advantages: flexibility, in-depth penetration, opportunity for clarification, and relative 
low costs of materials. 

Disadvantages: tirne-consUming, personal contact is potentially threatening, and 
responses are potentially highly reactive and subjective. 

. Qraf Session This instrument, a group interview, is a structured period during, which 
responses tc specific training matters are elicited either formally or informally from a 
trainee group. Given the complexity of dealing with more than one participant^ these 
sessions usually attempt to solicit only reactive data (i.e. opinions, judgments^ feelings). 
The responses are typically subjective, affective, and anecdotal. 
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Use Requirements: trainee cooperation ano interest, tim^, and a skilled leader. 
Advantages: immedlSicy, directness^ low cost. ' 

Disadvantages: leader or single member may influence other ^coup members; 

Qaestionnaire The questionnaire is a set of . written, items jsed to elicit data from 
respondents on specif ic aspects of training and/or .work. Questionnaire formats include 
open-ended (the respondent is totally free to construct an i^idividual response), the 
fixed-alternative (the respondent must select fromi ambng\ specified choices), or a 
composite of the two types. Open ended queries; are gener&l and narrative in form; 
fixed-alternative types lead to more organized ; spe-cific, and qiiantifiabre data, bike the 
Interview technique, the questionnaire may be designed \ to solicit REACTiON 
PERFORMANeE/APPLiCATION, and ORGANIZATIONAL RESULTS data. 

' : _ \ 

Use Requirements: skLHed interpreters, time, participarit', cooperation, and/or 
management cooperation (particuiariy when attempting to gauge organizational impacts). 

Advantages: reloti- ly low cost, respondent sets^ pace, honest responses increased if 
anonymity is guar: . . teed. If the open-ended fomat ii selected then\ response choices are 
unlimited; if fixe -alternative, then quick processing and standardization are possible. 

bisadvantar-es: If open-ended^ the responses are difficoit to process and subject ' to 
misinterpretation; if fixed-alternative, the responses 3re limited.. 

Partic. itory Techniques 1 

Participa nt contact Thi^ technique involves an agreement between trkihihg participants 
and the training staff (or among the trainir:g participani'. themselves) stating specific 
: training-related knowledges, skills, and/or altitudes, and a schedule or program outlining 
implementation. Learning application goals and/or performance modilication objectives 
are specific and job-based, which provide a valuable indication of PERiFORMANeE and 
APPLIGATION. \ 

Use Requirements: participant commitment. 1 

Advantages: participant directed and paced, potential enhancement of j motivation and 
on-the-job reinforcement. j 

__- ./ _ _ i 

Disadvantages: potential resentment of training staff monitoring and 1 follow-up, and 
potential organizational constraints to implementation. ! 

Self-assessment Self-assessment procedures are those designed to enab'le the training 
participant to judge her/his own application of learnings and/or performance change 
onrthe-job within a prescribed or agreed-upon time. Self-assessment may ^Iso be used in 
th^ classroom setting to gauge the participant's knowledge gain or accomplishment of 
learning objectives. This technique, therefore^ mayL be used to obtain data relating to 
both ACHIEVEMENT and PERFORMANCE/APPLICATION. 



Use Requirements: cooperative pa . .:ipants. 



Advantages: useful to training participonts, non-threatening. 

Disadvantages: requires self -analytical individuals, oata is normally unavailable to 
training offices arid is totally subjective. 

Organizv ^ uonrii Analyse s 

Organizational docum ents This technique refers to a analysis of an organisation's 
written records (e.g. memoranda, its reports, program jtSj production schedules'and 
reports, performance appraisals) whicn measure organizuaonai change due to training. 
The objects of such analyses are documentary indications of PERFORMANCE change or 
learning APPtiCATiON oa an individual basis or oh a, work ' unit basis 
(ORGANISATIONAL RESULTS)^ , s . 

Use Requirements: availability, of aqd-^ access to doc(iments. time, management 
cooperation, and translatability of documents. 

Advantages: object! /ity, reliability, job-based, pre- and post-training ccmpat-abjJi^ y. 

Disadvantages: possible subjectivity of preparer or transla^^ need for conversion to 
usable form since content of documents is not training specific. 

The following chart (Chart I) summarizes the relationship between cv3'u?;-^^ objectives 
and techniques, showing which techniques are used to evaluate which object--c.: 
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CHART I 

EVAhUATieN TECHNIQUES USED TO MEASURE 
EVALBATiGN OBJECTIVES 

f ' 

' ' EVALU- I I0N eBclECTiVES ' ' 



TECHNLQUES 


REACTION 


ACHIEVEMENT 


PERFORI-IANCE/ 


ORGANISATIONAL 








APPLICATION 


RES'JLTS 


TESTS 










Written Tests 




X 






- Performance Tests 




X 


X 














0BSERVATI0NS 










Non-verbal Indication 


X 








Observation 


X 


X 


X 














INTERR06ATIVE INSTRUMENTS 










Interview 


X 




X 


X 


Oral Session 


X 








Questionnaire 


X 




X 


X 




PARTICIPATORY TECHNIQUES 










Participant Contract ■ 




e x 


X 




Self-Assessment 












ORGANIZATIONAL ANALYSES 










Organization?.! Jocuments 






*x 


X 
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' Evaiaatibn Design Considerations 

Four basic scientific evaluation designs representing a highly sophisticated approach 
are drawn from: Zenger, 3. H., and Kenneth Harg.^, "Assessing Training Results: 
h's Time- to take:the Plunge!" T^^mniriR and Development 3ournal , January, 1982. 

The pre-test/post-test, control group design. This design has three fundamental 
requirements: 

b Use the same measurement before and after training; 
b Measure both a trained and a control group; 
o Randomly assign people to the trained and control groups. 

If these requirements are not fulfilled, then rival explanations may be offered 
subsequently to account for any obser-. :d changes. 



b- The post-test only, control group design. In this design neither the contro' ^r 
trained group need to be measured before the training. The pre-trau; ;^ 
measure require: d in the previous design helps guarantee that the control a.iJ 
trained groups are comparably composed. If equivalence between gro is 
reasonably assured, their comparisons can be made reliably on post-training 
scores alone. 

c. Repeated testing of the same group. This design involves testing the same 
group numerous times during both training and non-training periods. When the 
tests are administered frequently duri^ng both periods the e valuator can say, 
with increasing assurance, that extraneous factors are not responsible for 
changes. 

d. The pre- and post-tests of trained groups design. This design reUes on the same 
p»rini-Hle as th- previous method. Involved are a number of pre- and post-te:sts 
of thjr group performed at different times in the same general setting. 

"Comparing ail these procedures, it is the pre-test/post-test, control, groap design 
that provides the most unambiguous evidence of training effectiveness." (Zenger 
and Hargis, pg. i^) 
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TABLE I 

FACTORS TO CONSIDER IN THE DESIGN OF TRAINING EVALUATION 



0 Who took the course? 

0 We e the objectives met? Did the course work? 

^FORMATION o Was the course methodology and design appropriate? 

0- What factors intev^ered with the consistency of course 
: presentation or with intended learning? 



b The participants "in training, and the control group 
d Instructors and administrators of training 
0 Job performance evaluations (e.g. supervisors, 
colleagues, subordinates) 
[dVIDERS 0 Independent observers (e.g. contractor) 

0 External experts 
0 Training staff 



STkUMENTS 



^ING 



IFIDENCE 



0 Interview 

0 Non-verbal Indication 

0 Observation 

0 dralj^ession 

0 Organizational Documents 

0 Participant Contract 

0 Performance Test 



d Questionnaire 
0 Self-Assessment 



b Before trc:ining begins 

d n^ring training. 

0 immediately following training 

d Ldng after training 

d Once 

0 Periodical ly 
0 Anytime 



0 PropeV sampl ing or measurement of entire trainee 

population 
0 Expert face or content validity 
0 Correlation with related standards or measures of 

perforrance 

0 Research designs to control unwanted variables 

0 Reliability/validity measurements of test instruTients 



imii 



mjme 



DBIRMI3 



Nanes, titles, offices 


Participants . 


Sigiv-in sheett^ 


Before a after 




classificaticn; years 


training or 


survey/qiEStisifiaire 


trainiitii 


pcpulation. 


in vxatioi, [lior 




persomei records* 


Data need be 




Who took the cmrse? trainirq/ age, raoe/ 


staff. 




cbtained oily 




sex. 






oiDe oniess 










OEsiDgr^c Changs 










is a factcx in the 










evctoticn 





Hbre the objective 
net? 



Did tiia uxjrse 
wcrk? 



Perfonnaire of skills 

Recall of irfonratlch. 
Recognition of infomo- 
ticn. 



Pcst-test After training 

otservers, ccntrol Pre- ardPcsb - Before and after. 

grdp, job Perfomaice b^t, Datacan.be 

performnoe written tests, obtaiied axe, 

evaluatioos. questicnnalre, periodically or ^ 

interview, arytiite. 
. ofceervaticn. 



Proper sar^ing^ 
research design, 
e}^t assessieit, 
rocrelation with 
related jsrfonranoe 



reliabilily/valii 
of test. 



MS tfe coarse 
nethodoiogy and 
feign appro- 
priate? 



Pas, effcrt to leai'n, 
quantity of fresaitatian 
vs. practioe/ocerciss, 
piaoerEnt at exercises, 
dcptJi ctf informaticn, 
level (e.g. elementary 
advanced) , utilitv of ref- 
erence neterials or job 
aids, utility of course, 
quality oi" discussion. 



Participants, 
instriEbor cr 
cdiiirdstratDr, 
observer. 



QosticiirHire, 
interviws, ccal 
sessicT, te- 
raticn. 



Daring or 

initi&liately 

jfte^train'jig . 

Data^ieedbe 

dbtaimdctty 

cr^e. 



J3p:t assessiresiti 
peeper saipling. 



y#at factors inter- 
fered with the oufii^ 
tency of course 
presentedch or 
with iiitetifed learhlh^? 
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(Quality of facilities, 
quality of pra\r 
ni}di'/ii?':.''L'iir:.s, 
instr aisis 
in Hk office dr 
agency, inaoiirateoc 



Participts, 



instriictDrs. 



psticrnaire, 

interview, 

cbservatich. 
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Dlirli^ ci: 



after training . 
Data re©3be 
cbtaired 
only cnbe. 



Proper 



ho 
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APPENDIX B 



Achievement— -The degree to v/hich 1^- -jnd/.)!" [0o.rriiP2 objectives 

are accomplished. (See Learning-) 

Application— The implementation or use of . i.ecJJ V'.. hg upon return to the 
workplace. (See Behavior and Performance' 

Battery — A series of related tests administered together. ^ 

Behavior— An assessment of individual performnncs on the job, (See Data.) 

eiient— -The person(s) or organization(s) for which an evaluation is 
conduct.- d, ^ 

Control Group— A group of persons who do not participate in the training 

program but who are similar in all relevant respects to those who do participate* 
Control groups are used as a basis for comparison. / 

Correlation— A statistical process which shows the degree to which two o/ 

more events or objects are related to each other. In evaluations of training program 
effectiveness, cdrrela^idns may be calculated to determine the relationships amiong 
-factors effecting training results. / 

Criterion, Criteria— Weasu^ $ of trainirg effectiveness which ref lectv^he 

goals and objectjyes oi the training pro^gram,^ They 'provide a d^scripti^on or image of 
what should happen, thereby facilitating comparisons between what should have 
happened and what did happen^c'^ / 

Data — Factual material from which conclusions may be drawn. AVhen evaluating 

training, four categories of data may be obtained: Reaction, Achievement, 
Performance/Application, and Organizational Results/ / 
_ _ _ ... I ^ ^ - 

Design— A strategy which the evaluator uses tc collect data. The' design 

' usually specifies who wilj'be measured (experimental group, control group)^ and when 
they will be measured (pretest^ p.osttest). The purpose -of the design is to guaid 
against the possibility that something other than the treatment: causes the observed 
effects of the" training program. 

-- - _ - _ - - -- - - ^ / - - - - -- 

Evaluation— A deliberate process which provides specific reliable information 

about a selected topic, problem, or question for purposes of determining value 

and/or making decisions. 

/' 

Experimental Design— A data collection strategy which attempts to control as 

many -elevant and irrelev ..ut variableis as possible. An experimental design grains its ■ 
ri^or by using control groups and random selection and assignment of individuals to 
groups. 

Experimenta[ Group— A_group c^f subjects who receive the experimental 
tr^tment'in a design. . , ' 
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Forced Chbice^A special kind of multiple-<:hoice item whic the 

respphdeht to choose the more descriptive of two or more equally attractive or 
Unattractive statements^ This type of item is more often used in personality .and 
attitudinal measures when social desirability of the possible responses may interfere 
with the selection of the alternative chosen; 

Formative Evalu^rion— The process of judging an instructional package or 

nrocess or it' components during the developmental period for the purpose of 
providing persons directly involved, with the formation of tl^o entity with feed-back 
as to possible improvements. (See Evaluation,), 

Historiai Data— Data collected by the organization as part of its ribrmal 

day-to-day functioning. It can • include numerical indices such as absenteeism, • 
turnover or production rates^ and organizational documents such ^s agency 
memoranda, audit rcportsj program budgets, employee rating forms, supervisor 
appraisals, and written plans. * _ 

Hypothesis— A statement proposing a plausible relationship between two or 
more variables^ 



Learning— The principles, facts, skillSj and attitudes that participants gain 
from training. (See Achievement-) ; 

Matching— A pairing of subjects on the basis of background information 

factors such as age, level of education, or organizational status, followed by random 
assignment of one member of the pair to the experimental group and the other to the 
control group. This process, used when totally random selection and assignment are 
impossible, helps prevent the personal characteristics bi the subjects from 
contaminating the evaluation results. 

Open-ended— A question allowing respondents to answer freely in their ov/n 

words rather than restricting their ; answers to a few stated alternatives as in a 
multiple-choice auestion. Although they are more difficult to analyze than 
multiple-choice quii^tions^ open-ended questions allow for a wider variety of responses. 

Organizational Results—The impact training dn.the organizaLion or job ^ 
* environment. 

PAPA (Participant Action Plan Apprd?ch)— An QP^N/l-d^^vejoped approach intended : 

to facilitate and analyze the iec-rnlr.g usaj',e of trainees on-the-job. The approach 
includes five steps: planning, in-course activities, toilow-up activities, analysis, and 
report. ' ' 

Perforrnance— The group of behaviors and/or applications copslituting an ; ^ 

individual's, group's, or organization's work actions. (See Behavior and Applicatiohc) — 

Posttest, Postcourse, or Posl:training>\easurement— A measurement taken after 

the iraining- program ha5 ehdedi The resulting information can be used to deter^nine 
whether qr net trainees nave achieved training objec-' yes. If compared to a pretest 
the posttest provides a measure of change probably attributable to training. 
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Pretest, Precoorse, or Pretraining Measuremeht^A measurement taken before 

the training program begins or during its early stages. The resulting information may 
-be used to provide jnstructionai design with a picture of/ the skills and abilities of 
the average entering trainee or may be used to give the instructor an idea of how 
much the group already knows r^elative to the learning objectives. A pretest also 
provides a baseline for comparison against a posttest. <, 

Procedures— Instruments, devices used to obtSn data for evaluation. 

Random Assighment or Selection— The seiection'of cases or subjects in such 

a way that all have an_equal probability of being included, and the c! ^ctidn of one 
• subject has no influence on the selection of any other subject. 

Rating— The process of judging sorneone or something according to 
predetermined criteria. (See Scale.) ^ 

Reaction— An indication of how well the * * : .ees liked a particular program, 

including materials, instructors, fa . ' , methodology, content, etc> (See Data.) : • 

Renability— The degree to which a dev^ istrument (procedure) \ 

.measures a given characteristic con. Jy, 

Rigor— A term used to describe the a^nount of control exerted by a design . _ " . 

and the consequent precision that can be attributed to the find'ngs. The more 
rigorous the design^ the greater is the level of confidence that one can have, in the ■ 
findings of an evaluation. 

Sample— A subset of the population, usually selected to be representative of 
the whole group being studied. 

Scale— A graduated continuum which allows a rater to assign numerical values - : 

ranging from low to high to a given trait or characteristic. Scales generally have 
between three and nine categories which may or may not have accompanying 
descriptive adjectives or numbers. ' . ; " 

^ ^ Significant, Statistically Significant— A statistically significant_event is 

one that has a J;ow likelihood of happehing by chance. Significance does not mean 
irnportance; it merely means that a difference, such as the difference between the 
scores of two groups on a postest, was due to some difierence between the two 
groups rather than due to chance. . ' ' ' ^ , 

Subjects— Individuai.3 selected to participate in any facet of a design. 

Summative Evaiua uicn— The process of judging a completed instructional 

package or process for the purpose of providing^ the end User with information as to 
its demonstrated effectiveness in a given .situation. Based on this information^ the ^ 
user may docide to purchase the entity (if a potential user), or retain it (if a current 
User). (See aiuation.) > 

Treatment— The training program or a variation in the training program given 
to an experimental group in a. design. 
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\ Test-.A serie^ of questions, exercises, or other means of measuring the * . . 

\ 'kn5wfedge, skills, abilities, or aptitudes of an individual or group against selected 
\^ criteria or hbrrhs* 

Validity— The degree to which a device or ihstrurr^en . -edure; measures ^ 
\ wHat it was intended to measure- 

Variable-^Sbrhething that is capable of changing in value over ti^ne. One . r . - - 

. \ purpose of a design is to control for (limit the Yariability of) irrelevant variables so 
\ that jthe effect of relevant variables may be observed. , / 

\ . . ' ■ 

\ ■ - 
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(Discussion between two preeminent professionals about the training 

office's role in performance change and how to increase organizational 
influence) f 

Brethower, K.S. & Rummler, G.A. Evaluating training. Training and 
Development Journal , 1979, 33{5) y 14-22. 

(Presentation of a framework for viewing evaluation alternatives and 
deciding what evaluation is appropriate, by using a general systems 
model; evaluation levels, considered deal with participant -satisfaction 
with course, training's accomplishmfints of goals » participant use of 
concepts, and negative or positive impact on organization; also 
includes guidelines on conducting evaluation studies with a matrix of 
questions to ask for each evaluation level, and suggested evaluation 
designs) ' . 

Brinkerhoff , R.O. Making evaluation more useful. Training and 
Development Joar4taJ^ , 1981, 3i-(12), 66-70. 

(General discussion of various purposes of and uses for evaluation 
information, including questions which evaluation studies can answer 
during planning, delivery, and follow-up/feedback phases of training) 

Brown, M.G. Evaluating training via_multiple baseline designs. 
Training— and Development Journal , 1980, .34(10) , 11-16. 

" (Drscussion of the multiple baseline design , whlch^is adaptabie-to 

most training research, and provides a convincing demonstration of 

functional relationships between training and improvements in employee 
performance) 

Buchanan, P.C. A system for evaluating supervisory development 
programs. Personnel , Jan. 1955, 335-347. 

(Discussion of past methods used for evalttation and questions to 
systematically answer in order to evaluate supervisory training 
programs) 

Campbell, J. P., et al. Ma^ nag e ri ai beha vior, performance, and 
effectiveness . New York: McGraw-Hill, 1970. 

(Chapter on Methodology of Training J^valuation summarizes general 
designs for evaluating management trc-lning and provides practical 
guidelines) - ^% 



designs f or research . Chicago: Rand McNally, 1963. — " 

(Frequently-ased reference for basic research designs in social 
science .settings; certain designs lend themselves well to training 
evaluation efforts) i-i.dj.uj.ng 

♦ 

Chabotar, K.j. & Lad, t.J. Evaluation guidelines for Xralnitte 
^ro|rams. Lansing. MI: Michigan Dept. of Civil Service (Dr. Donald 
J. Willis, Project Director), 1974. Developed for Midwest 
^ Intergovernmental Training Committee, Intergovernmental Personnel 
Program, U.S. Civil Service Commisslbn. 

(Illustration of the evaluation problems confronted by training 
offices and guidelines for solving them; the book familiarizes the 
reader with criteria that should be met in order to achieve an 
effective evaluation and with evaluation designs and sample forms) 

c 

element, R.W. Evaluating the effectiveness of management training: 
Progress during the 1970's and prospects for the 1980's. Human 
^eswrce Management . Winter 1981, 8-13. 

(Reviews types of studies done to assess effectiveness of management 
training and general study outcomes) 

Coffman, L. Successful training program evaluation. Training and 
Dev etopm ea^ Journal . 1980, 34( 10) , -84-87. 



(Brief description of an evaluation process using "key clients" - 
usually the supervisors of trainees - to collect data on training's 
impact -on-job-performance and to share findings in a group Setting) 

Cornwell, J. B. Justify your training efforts with on-the-jo'b 
performance. Trai n ing/HRD , 1980, j7(8) , 34-36. 

(Provides post-training survey iristruraints to 3e sent to participants 
and their supervisors, designed to determine extent of performance^ 
change and the reasons for the amount of change; also suggestions for 
more general follow-up questions if survey instrument ts not feasible) 

^^mlng, B.S. A systei for evaluating training programs. Personnel, 
Nbv/Dec. 1979, 33-41. - 

(Ten steps to 'determining which training programs are worth 
evaluating) 
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Doming. B.S. E vaquatJJtg^^ajLning progr ams; A guide f-Oj^aintng^ 
^Sner ." Englewdod Cliffs, NJr frentice-Hall. & Wash.. DG: American 
Society for Training and Development, 1982. 

(Discussion of the skills needed by an evaluator of training programs, 
as well as a discussion of evaluatibn techniques to be used at 
different points in the training process and for different purposes 
of evaluation) 

Elfcins, A. Some views on management training. Personnel Journal, 
June 1977, 305-306, 311. 

(Summary of evaluation of a basic management development pj_ §ram and 
the issues raised by the outcomes, such as the drop in confidence in 
ability; prt- to post^course. and the role of attitude about the 
course in changing behavior) 

Fl-^hree A R S Howe. C An evaluation of training :;in three acts. . 
Tralni^;ind:^ve^oPment Journal , 1977, ^(7,8,9). (Three issues.) ^ 

(An example of the process for planning and implementing a t-raining 
evaluation) 

Fast, D. A new approach to quantifying training prQg^^^^ 
effectivreness. Training and Development Journal , 1974, 30(9), « 14. 

(A method for end^of^course evaluation, based on^partlcipant selection 
(at the beginning of training) of personnally relevant course 
objectives) 

Franklin, Jr., W.H. Why training fails. Admlnis t^atlA^JManagement , 
July 1931, 42-43. 72-74. 

(Reviews reasons for ttra-'ning efforts not being a-tcessful and 
suggests methods for overcoming problems) 

Georgenson. D.L; The problem of transfer ^calls for partnership . 
Training and DevelapMeni^Jonrnal . 1982. 36'^10). /3 /o. 

(Discussion the problem of transferring the knowledge skills 
Ic-arned in the classroom to the job. and suggestions for doing this in 
an effective manner by obtaining management support) 

Gilbert. T.F. Training: The $100 MHion opportunity, fcaining atA 
Development JQur-nal . 1976. W(ll) , 3-8. 3. 

(General discussion of appropriate^ways to describe costs of training. 
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Gilbert, T.F. Measuring the iSotentl a t for performance Improvement, 
Tralnlng/HRD, 1978 , J^C 1 2) r 25-28, 



(Summary of the concepts of worthy performance, behavipr vs. 
accbm^llshment, and potential for improving performance; excerpted 
from Gilbert *s book. Hu m a n Co iapetence) ■ ^ 



Gilbert, t.F. A question of performance^ _ Par t I : The PROBE model* 
Training gnd n^v^tppment Journal , 1982, 36(9), 2 1-3(J. Part * : . 
App^lying the PROBE model. Training and Development Jbwti^i , 1982,? 
^(10), 85-89. . ^ . - 



("Profiling behavior" - PROBE - consists of a set of questions to ask- 
about work environments in order to determine barriers to exemplary - 
performance; the articles describe this performance analysis approach 
and provide an example of its use) 



Goldstein, I.L. Training: Program developaeiU^atid-evaluation . 
fonterey, CA: Brool '=«/Cole Publishing Company, 1974. 



(This book describes a systematic approach to designing, developing; 
implementing, and evaluating training programs, as well as discussing 
learning theory and instructional approaches) 

Gbldstein, I.L. The pursuit of validity in the evaluation of- training 
programs. Human Factdrs^ , 1979, ^(2), 131-144. 

(Discussion of questions such as', "Did training make a dif f erehce in 

this situation?" and "Do results in this situation apply to/ other 

trainees or othiBr of ganizattons? " ) . : _ „ _^ /v 

Hamblin, A.C. £valuat:±an,^nd control of training . London: 
McGraw-Hill Book Company (UK) Limited, 1974. 

(Discuasion of alternative evaluation methods, including descriptions 

strategies, objectives, and techniques; examples cover the range 
fr^m cost-benefit approaches to reaction measures) 



Heydihger, Jr., R.B. Planning an innovation: An inventory of decision 
variables. Xmprovlng^itumari Performance Quarterly , 1979, S(2), 
123-133. 

(A checklist of characteristics useful for planning a program 

evaluation study, such as time frame, reference points , methodology , 
and feedback) . 
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Kearsley^ G. Cbsts» benefits ^ arid productivity In tralhlnjg systems , 
Reading, MA: Addisori-Wesley , 1982. ' 

( A"dbwri-to-earth explanation of four models - two related to training 
costs; one to short-terra benefits of training; and one to training and 
prbducbiyity ) ^"^^ 

Kirkpatriclc, D.L, Evaluation of training. In R.L.^ Craig (Ed.)^ 
graining and development handbook (2rid ed.). New ^ork: McGraw-Hill, ^ 
1976. 



(Four types of training ievaluatibri data — reaction, learning, 
behavior, and results - are discussed) 



Korb, L.b. How to measure the results of supervisory training. 
P^rsotmeJ^ , March 1956, 378-390. 

(Guidance on various levels of supervisory training evaluation — 
effects on participants while in training and after training, and 
effects 6n the organization) • 



Leonard, Jr., E.<]. Assessment of training^ needs . Fort Wayne, 
IN: City of Fort Wayne (Jerry D. Boswell, Project Director), 1974. 
Developed for Midwest Intergovernmental Training Committee ^ 
Intergovernmental Personnel Program, U.S. Civil Service Commission. 

(A training manual which includes examples of methods for doing 
organizational i job, and manpower analyses , and identifying training 
needs) *^ 

■ I . 



Meier, T/K. & Pulichene^ J. P. Evaluating the effectiveness of 
asse'rtiveness training. Training and Development Journal , 1980^ 
^(2), 66-68. 

(Describes a tnethbd for determining extent to which employees exhibit 
"assertivie '\ behavlb^ , as j udged by the eiiiplbyeesl supervisors; 
judgEcerits made from the beginning' of training through a period after 
training cdrapletion) " " « 



Mezof f , B. How to get accurate self-repbtts of training outcomes. 
Training and Development-. Jou^rnaL," 1981 , 3^(9), 56-61. 



(A miethbd fbr Increasing the accuracy of participants' judgments of 
their knowledge and skill levels before training, cojnpared to after 
training) 
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Mlnlck, R.p. fit Medlin, S.M. Anticipatory evaluations in HRD _ _ 
prograramingi Training and Development Journal , IQSS'J j7(5) , 89-94. 

(Describes in general different types of evaluations to be done during 
diiferent phases of the training cycle , par ticularly the collection of 
information while planning HRD programs) 

Mbrrisey, G.L. & Wellstead, W.R. Supervisory training can be 
measured. Training and, Development Journal ^ 1980^ 34(6) ^ 118-122. 
(Originally published June "1971.) 

(Use of objective-setting at the end of training, and progress reports 
after trainings as a means of evaluation and participant commitment) 

Mbseli J,N. Why training programs fail to carry over. Pe rsonne l , 
Nov. /Dec. 1957, 17-25. 

(How to (ensure that what participants learn is transferred to the job) 

Parker, f.C. Statistical methods. for measuring training results. In 
R.L. Craig (Ed.), Training --and dpfvei:apm&nt handbook (2nd ed.). New . 
York: McGraw-Hill, 1976. 

their application to effects of training on 
individual and organizational performance ^ 



Peterson, R.O. (Ed.) ASTD Research Series No. 3; Determining the 
payoff of management training . Madison, WI: American Society for 
Training and Development, 1979. 

(Papers from the first annual invitational research seminar ,. covering 
topics such as general guidelines for evaluating" the outcomes of 
management training and an approach to determining the value of such 
training) ' 

Phillips, J.J. Handbook^^ f training e:vaJ. u a tion and jneaaur-gment- 
methods . Houston, TX: Gulf Publishing, Company, 1983. 

(Covers a wide range of training evaluation areas, including desiguiag 
ail evaluation effort, selecting_instruipents and sample participants, 
collecting and analyzing data, determining program costs, measuring 
return-dn-investment ) 

Putnam, A,.b; Pragmatic evaluation. Training and Development Journal ^ 
1980, mXiO), 36-40. ' ' ^ 

(Gtennral dtscicission of problems with evaluations which are . 
"truth-seeking;" lis ts"^ eight questions to guldis evaluatbr through 

" isrii&mafi n" Acsf^ptssment efforts^ 1 Q C: 



(Measurements methods and 
personal characteristics, 
and return-on-investment) 
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Rummli-r^ G.A. Yoii need performance, not just trainlngi Training , 
1977, ildO), 50-53. 

(Summary t)f the causes of poor performance and the role of training in 
managing performance 'improvement) 

' Salinger, R^D. & Deming, B. Practical strategies for evaluating t 
training. Training^atxd Developmen t Journal , 1982; , 20-29. 

(Description of evaluation strategies which answer suchquestions as 
the appropriateness of the training and the transferability of the 
training to the job) ^ 

Smith, M.E. An tliustratiou of evaluating post-training job-- ; 
performance, Improving Human Performance Quarterly , 1979,- 8(3), 
181-201. (a) 

0 - - -: - 

(A study of a new course for telephone installers, compared to an old 

one; a variety of measures were used to compare recent graduates of 

the two courses as well as performance of experienced installers; also 
an analysis of the evaluation measures themiselves) 

Smith, M.E. Exchanging ideas on evaluation: 15. New England 
Telephone *s training management operational review plan. NSPI 
Journal, 1979, 28(6), 4^-47. (b) 

(Description of New England Telephone's quality control process for 
its training function, including a m^itrtx of 34 variables measured, 
(e.g. ,_ timeliness of training, efficiency of develbpment) and the 20 
: general types of exercises used to measure variables) 

Smlthj M.E. Evaluating training operations and programs. Training 
and Develbpment JoaraaJ^ , 1980, ^(10), 70-78. ' ^ ' 

(Reviews Brethower/Rummler evaluation nodel and cpmbihes with AT&T 
generic list of major training organiza.tion activities; 4iscusses 
problems in doing evaluatiotv and conditions supportive to evaluation) 



Smith, M.E. Fiuld research desij^ns.' NSPI Journal ^ 1982, 24(2), 
27-29. ^ ^ ' ' ' 

(Reviews "quasi-experimental" designs which can be used to conduct 
evaluation studies; gives references to efforts in which designs were 
useH) 
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Snyder, R.A. , Raben , C.S., & Parr, J.Li A EQodel for the systematic 
evaluatibn of human resource develbpnierit programs. Academy of 
Map- n t Review , 1980,; ^(3), 431-444. 

(Discusses different types of evaluation appropriate at different 
stages of developing and implementing an HRD program, and discuases 
^ objectives and guiding principles for each type of evaluatibn, e.g., 
summative evaluation for assessing prbgram effectiveness and 
efficiency) 

Spitzeri D.R. But will they use training on the job? Traihing/HRD ^ 
1982, m9), 48, 105. ' o 

([Brief descriptions of twelve t«^chniques which will help transfer 
learned classroom skills to the job; examples are personal action 
planning, group action planning, the buddy system, and follbw-nip 
sessions)- 

Thtagarajan, S. The ^President ' s Pages: Transitional evaluation - : 
Correcting and confirming planned change. JJSP!L^Qwna3^ , 1979, X£l(3) » 
1-3. 

(A general approach for comparing the costs and benefits of old and 
new systems, " e. g. , different tnethods of training, with alternatives, 
depending on what kind of data is available) 

Thomas, E.A. Training needs identification: A turning point. NSPI 
Journal , 1982, ^(8), 6-8. 

(A performance-based process which can be used to evaluate the 
adequacy' of the organization's training progran and to anticipate 
training needs; an example of a question L's: Wliat type of new 
employees, i.e., with what skills, are being recruited and will be 
recruited in the future and for what pbsltions?) 

Training Research Forum_members . The emperor's clothes.. Training and 
Development Journal , 1979, 35(6)^ 50-60. (Originally published July 
1970.) 

(Describes three basic abilities underlying management behavior in any 
settingand stressesthe need to inyblve . tbp roahagemeht in the design 
and implementation of a management development program) 

Warren, M.W. Pellvering_ training in a decentralized organization. • 
Training/HRD , 1978, L5(12), 39-41. 

— ^(Consequences bf training tb participants, and their supervisors and 

managers^; hbw supervisbrs can ehsu£e_tjiat performance changes occur as 
a result of training) ; ^ — =^ 

.... . ■ . • • ■ 2\ju : ■ ^ 

ERIC 



Warren, M.W. Tralftltig^ f or resait a;_A. aystfems approac to th e 
development of ^hanxan- resources In industry C2nd ed,). Readings MA: 
Addi son-Wesley, 1979. \ ' ' 

(A book about the systems approach to training which looks at ttairiirig 
as a technology (an applied science of human performance) and designerd 
with a specific mission tn^mlnd; some concepts discussed are: training 
as a system, estimating (josts, evaluating training actions i and 
developing management training) 

I ■ * _ _ _ _ 

Wise, R.E. & Zern, K.R. Identifying and improving management skills 
at Hartford National' Bank.. Training/HRD , 1982, L9(ll), 56-58. 

(Description of ttie procedures used at a bank :b trair senior managers 
in basic interpersonal skills through a behavi >r mode Lrig program; the 
process tncioded a needs aissessment arid an eva uation using a 
pre-/post-test and control group desigri) 

Zeflce, R.. Management training and development: Measuring the impact. 
Training ,' 1977, }MlO) , 62t64. (a) 

(Brief coimnents by HRD managers and others on issues to consider 
tn evaluating management trairiing) 

Zemke, R. Task analysis: Figuring out^what people nev»d to learn. 
Training , 1977, _li(12), 16-20. (b) 

(Briefly reviews task analyses techniques ; f ami:iiarizes reader with 
terminologyand processes of, for example, critical-incident 
technique j flow-charting, focus groups) 

Zerafce, R. & Kratnlinger, T. Etgui^jLng things out: A trainer's guide to 
needs and task analysis . Reading, MA: Addison-Wesley , 198?. 

(Extensive coverage oT vaf ibias^^ t for studying 

organizations, perforratng task analyses, and determining training 
needs) 



Zemke, R. , Standke , t. , & Jones, P. (Eds.) Designing ^and deliv e r i n g 
cost-ef fggtive trafging - arid measuring the results . ^Minneapolis , 
MN: Lakewood Publications, 1981. 

) , t. . . . _ . __: 

(Collf'ction of articles oa wide range of training topics, from the 
last 16 years of Training magazine) 
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Zehgerj J.H.^S Hargis^ K,' Assessing training results : It * s time to 
take the plurige! Training and Development "Jourrial , 1982, 36( IQ 
11-16. • . ' . : 

(Discussion of why training should be evaluated and some guidelines t \y 
and general methods for doing itj includes examples of desi'gns to 
measure on-^ob behavior change of supervisors) ' . 
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